Sep 17 00:30:02.536551 ip-10-0-143-51 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 00:30:02.948873 ip-10-0-143-51 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 00:30:02.948873 ip-10-0-143-51 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 00:30:02.948873 ip-10-0-143-51 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 00:30:02.949351 ip-10-0-143-51 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 00:30:02.949351 ip-10-0-143-51 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 00:30:02.951105 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.951021    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 00:30:02.955271 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955257    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:30:02.955271 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955271    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955275    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955278    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955281    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955290    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955296    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955300    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955303    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955306    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955309    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955312    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955315    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955318    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955322    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955325    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955328    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955331    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955334    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:30:02.955329 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955336    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955340    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955343    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955345    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955348    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955350    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955353    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955356    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955358    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955361    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955363    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955366    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955368    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955371    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955374    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955376    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955379    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955381    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955383    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955386    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:30:02.955752 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955389    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955391    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955393    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955396    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955399    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955401    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955403    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955408    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955411    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955414    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955416    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955419    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955422    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955424    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955428    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955430    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955433    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955436    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955439    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955442    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:30:02.956222 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955444    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955447    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955449    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955452    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955454    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955457    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955459    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955462    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955464    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955467    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955469    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955472    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955476    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955478    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955481    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955483    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955486    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955488    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955491    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955493    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:30:02.956757 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955495    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955498    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955500    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955503    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955505    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955508    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955511    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955514    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955516    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955519    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955521    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955524    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955526    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955529    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955531    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955533    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955536    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955538    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955541    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.955544    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:30:02.957231 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956258    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956266    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956270    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956275    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956279    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956283    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956286    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956289    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956291    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956294    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956297    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956301    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956304    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956307    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956312    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956314    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956317    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956320    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956323    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:30:02.957751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956326    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956329    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956332    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956334    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956337    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956340    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956342    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956345    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956347    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956351    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956353    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956356    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956359    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956361    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956364    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956366    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956369    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956371    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956374    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956377    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:30:02.958209 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956379    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956382    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956384    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956387    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956389    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956392    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956394    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956397    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956400    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956402    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956405    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956408    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956411    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956413    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956416    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956418    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956421    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956424    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956426    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956429    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:30:02.958748 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956431    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956434    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956437    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956439    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956442    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956444    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956447    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956449    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956451    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956454    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956457    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956459    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956462    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956464    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956467    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956469    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956472    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956474    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956477    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956479    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:30:02.959258 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956482    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956485    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956487    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956490    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956492    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956495    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956498    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956501    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956503    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956506    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956508    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956511    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956513    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956516    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956518    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956521    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956523    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956526    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956528    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:30:02.959751 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956530    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956885    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956889    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956894    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956898    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956901    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956905    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956908    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956910    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956913    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956915    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956918    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956921    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956924    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956927    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956930    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956934    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956937    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956940    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:30:02.960200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956943    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956945    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956948    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956951    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956954    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956957    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956960    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956962    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956964    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956967    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956969    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956971    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956974    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956976    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956979    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956981    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956983    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956986    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956988    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956991    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:30:02.960687 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956993    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956996    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.956998    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957001    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957004    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957006    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957012    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957014    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957016    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957019    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957021    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957024    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957026    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957028    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957031    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957033    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957036    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957038    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957041    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957043    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:30:02.961156 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957045    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957048    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957050    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957053    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957056    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957058    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957061    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957063    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957065    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957068    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957071    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957073    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957076    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957078    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957081    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957083    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957089    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957091    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957095    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:30:02.961755 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957097    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957100    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957102    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957105    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957107    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957109    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957112    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957114    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957117    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957119    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957121    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957124    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957127    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957130    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957133    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957136    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957139    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957141    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957144    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957146    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:30:02.962221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957149    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957151    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957224    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957278    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957285    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957290    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957300    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957303    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957307    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957312    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957316    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957320    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957323    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957326    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957329    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957332    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957335    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957338    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957341    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957344    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957348    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957351    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957355    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957358    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 00:30:02.962723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957361    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957365    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957368    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957371    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957374    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957377    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957380    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957383    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957386    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957389    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957393    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957395    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957398    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957401    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957404    2576 flags.go:64] FLAG: --enable-server="true"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957407    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957412    2576 flags.go:64] FLAG: --event-burst="100"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957416    2576 flags.go:64] FLAG: --event-qps="50"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957419    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957423    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957426    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957429    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957432    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957435    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957438    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 17 00:30:02.963324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957440    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957443    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957446    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957450    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957453    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957455    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957458    2576 flags.go:64] FLAG: --feature-gates=""
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957462    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957465    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957467    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957470    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957474    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957476    2576 flags.go:64] FLAG: --help="false"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957479    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-51.ec2.internal"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957482    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957485    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957488    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957492    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957495    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957498    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957501    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957504    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957507    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 00:30:02.963954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957510    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957514    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957517    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957520    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957523    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957526    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957529    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957532    2576 flags.go:64] FLAG: --lock-file=""
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957535    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957537    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957540    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957546    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957548    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957551    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957554    2576 flags.go:64] FLAG: --logging-format="text"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957557    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957560    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957563    2576 flags.go:64] FLAG: --manifest-url=""
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957566    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957570    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957573    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957576    2576 flags.go:64] FLAG: --max-pods="110"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957579    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957582    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957585    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 00:30:02.964547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957588    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957591    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957594    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957597    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957603    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957606    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957609    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957613    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957617    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957622    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957625    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957628    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957631    2576 flags.go:64] FLAG: --port="10250"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957634    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957637    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09aa4da9941b65aff"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957640    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957643    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957646    2576 flags.go:64] FLAG: --register-node="true"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957649    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957652    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957655    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957658    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957661    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957664    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957667    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 00:30:02.965160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957670    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957673    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957676    2576 flags.go:64] FLAG: --runonce="false"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957679    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957682    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957685    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957687    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957690    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957693    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957696    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957699    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957702    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957704    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957707    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957710    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957712    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957717    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957721    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957726    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957729    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957732    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957737    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957739    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957742    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957745    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 00:30:02.966192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957748    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957751    2576 flags.go:64] FLAG: --v="2"
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957755    2576 flags.go:64] FLAG: --version="false"
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957760    2576 flags.go:64] FLAG: --vmodule=""
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957763    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.957767    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957910    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957914    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957917    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957920    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957922    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957926    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957928    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957931    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957933    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957936    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957938    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957941    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957943    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957946    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:30:02.967291 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957949    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957951    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957954    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957957    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957963    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957967    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957970    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957973    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957975    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957978    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957980    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957983    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957985    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957989    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957993    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957995    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.957998    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958000    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958003    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:30:02.967899 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958005    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958008    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958010    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958013    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958015    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958018    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958020    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958023    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958025    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958028    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958030    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958033    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958035    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958037    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958040    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958042    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958045    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958049    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958052    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958054    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:30:02.968524 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958057    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958060    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958062    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958065    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958067    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958070    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958073    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958075    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958077    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958080    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958082    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958084    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958087    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958089    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958092    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958094    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958097    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958099    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958102    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958104    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:30:02.969431 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958107    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958109    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958112    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958114    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958116    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958120    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958123    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958126    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958129    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958133    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958135    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958138    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958141    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958143    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958145    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958148    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958150    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958153    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958155    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958158    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:30:02.970097 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958160    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958163    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958165    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958167    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958170    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.958172    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.958885    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.967460    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.967478    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.968416    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.968434    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968526    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968534    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968538    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968542    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:30:02.970685 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968547    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968551    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968555    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968559    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968563    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968567    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968572    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968577    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968582    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968586    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968592    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968596    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968600    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968605    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968610    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968614    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968618    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968622    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968626    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968630    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:30:02.971333 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968634    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968639    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968643    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968647    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968651    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968656    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968660    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968665    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968669    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968673    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968677    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968681    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968685    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968689    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968694    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968698    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968702    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968706    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968710    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968714    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:30:02.972119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968719    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968724    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968728    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968732    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968737    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968741    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968745    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968749    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968753    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968757    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968763    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968769    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968773    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968777    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968782    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968786    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968790    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968795    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968799    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968804    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:30:02.972630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968808    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968813    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968817    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968821    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968825    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968829    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968833    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968837    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968842    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968846    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968850    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968854    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968858    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968862    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968867    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968872    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968876    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968880    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968884    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968887    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:30:02.973303 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968891    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968895    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968899    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968903    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968910    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968915    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968920    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968925    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968929    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968935    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968939    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968944    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968948    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968954    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:30:02.973788 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.968960    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.968968    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969138    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969145    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969150    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969155    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969159    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969163    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969167    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969172    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969176    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969180    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969183    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969188    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969192    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:30:02.974174 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969196    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969200    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969204    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969208    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969212    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969216    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969220    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969224    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969228    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969232    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969256    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969260    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969266    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969270    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969274    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969278    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969281    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969287    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969293    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:30:02.974564 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969297    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969301    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969305    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969309    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969313    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969317    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969321    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969325    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969329    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969333    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969337    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969341    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969346    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969350    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969354    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969358    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969363    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969367    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969371    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969375    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:30:02.975082 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969379    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969383    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969386    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969390    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969394    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969399    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969403    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969407    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969413    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969418    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969423    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969427    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969432    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969437    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969441    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969445    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969449    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969453    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969456    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:30:02.975596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969461    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969465    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969469    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969473    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969477    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969481    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969486    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969490    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969494    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969498    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969501    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969505    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969510    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969514    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969518    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969521    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969525    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969529    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969534    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969538    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:30:02.976365 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969542    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969546    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969550    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969554    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969558    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969562    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969565    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:02.969569    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.969578    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.969723    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.972758    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.973902    2576 server.go:1019] "Starting client certificate rotation"
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.974007    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 00:30:02.977060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.974044    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 00:30:02.977519 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.977505    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 00:30:02.978636 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.978617    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 00:30:02.993620 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.993603    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 17 00:30:02.996871 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.996855    2576 log.go:25] "Validated CRI v1 image API"
Sep 17 00:30:02.997845 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:02.997830    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 00:30:03.000509 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.000484    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 822bf7e5-789d-4ffe-9c7c-4c0a89914df5:/dev/nvme0n1p4 d527d08e-cc86-4e03-948d-c8581ad75121:/dev/nvme0n1p3]
Sep 17 00:30:03.000583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.000508    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 00:30:03.001377 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.001350    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 00:30:03.006863 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.006742    2576 manager.go:216] Machine: {Timestamp:2026-09-17 00:30:03.005289253 +0000 UTC m=+0.363791996 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100316 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d28a0b7243a8c34325852cb5e3794 SystemUUID:ec2d28a0-b724-3a8c-3432-5852cb5e3794 BootID:d2238a0e-d491-44fa-9853-137c3d25009d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:4a:c5:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:4a:c5:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:54:39:31:f5:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 00:30:03.006863 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.006854    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 00:30:03.006982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.006934    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 00:30:03.010432 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.010394    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 00:30:03.010573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.010435    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-51.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 00:30:03.011311 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.011300    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 00:30:03.011354 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.011315    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 00:30:03.011354 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.011323    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 00:30:03.012320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.012308    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 00:30:03.013605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.013594    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 00:30:03.019211 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.019194    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 00:30:03.019351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.019341    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 00:30:03.019387 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.019374    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 00:30:03.019387 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.019386    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 00:30:03.019444 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.019398    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 00:30:03.019444 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.019409    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 17 00:30:03.019444 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.019418    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 00:30:03.019932 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.019909    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 00:30:03.020004 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.019956    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 00:30:03.022664 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.022637    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 00:30:03.025575 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.025553    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qszrt"
Sep 17 00:30:03.026698 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.026677    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 00:30:03.026761 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.026713    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 00:30:03.028881 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028868    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 00:30:03.028937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028885    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 00:30:03.028937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028892    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 00:30:03.028937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028899    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 00:30:03.028937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028908    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 00:30:03.028937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028913    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 00:30:03.028937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028927    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 00:30:03.028937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028932    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 00:30:03.028937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028939    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 00:30:03.029151 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028945    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 00:30:03.029151 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028953    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 00:30:03.029151 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.028958    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 00:30:03.029890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.029877    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 00:30:03.029890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.029892    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 00:30:03.031466 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.031442    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 00:30:03.031538 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.031444    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 00:30:03.033926 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.033911    2576 server.go:1268] "Started kubelet"
Sep 17 00:30:03.033986 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.033938    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qszrt"
Sep 17 00:30:03.034033 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.033972    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 00:30:03.034113 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.034089    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 00:30:03.034150 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.034130    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 00:30:03.035702 ip-10-0-143-51 systemd[1]: Started Kubernetes Kubelet.
Sep 17 00:30:03.036912 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.036780    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 00:30:03.037430 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.037417    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 00:30:03.039513 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.039452    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 00:30:03.039920 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.039907    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-51.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 00:30:03.040940 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.039777    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-51.ec2.internal.18d5f444be18a65c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-51.ec2.internal,UID:ip-10-0-143-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-51.ec2.internal,},FirstTimestamp:2026-09-17 00:30:03.033880156 +0000 UTC m=+0.392382899,LastTimestamp:2026-09-17 00:30:03.033880156 +0000 UTC m=+0.392382899,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-51.ec2.internal,}"
Sep 17 00:30:03.043130 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.043112    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 00:30:03.043811 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.043795    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 00:30:03.043924 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.043907    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.044561 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.044540    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 00:30:03.044647 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.044549    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 00:30:03.044647 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.044580    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 00:30:03.044779 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.044698    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 00:30:03.044779 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.044705    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 00:30:03.045008 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.044991    2576 factory.go:55] Registering systemd factory
Sep 17 00:30:03.045044 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.045015    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 17 00:30:03.045336 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.045324    2576 factory.go:153] Registering CRI-O factory
Sep 17 00:30:03.045393 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.045338    2576 factory.go:223] Registration of the crio container factory successfully
Sep 17 00:30:03.045424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.045411    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 00:30:03.045462 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.045448    2576 factory.go:103] Registering Raw factory
Sep 17 00:30:03.045462 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.045462    2576 manager.go:1196] Started watching for new ooms in manager
Sep 17 00:30:03.045896 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.045882    2576 manager.go:349] Starting recovery of all containers
Sep 17 00:30:03.047514 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.047450    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:30:03.049830 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.049808    2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-51.ec2.internal\" not found" node="ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.059024 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.058030    2576 manager.go:354] Recovery completed
Sep 17 00:30:03.063015 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.063002    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:30:03.064976 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.064961    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:30:03.065060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.064988    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:30:03.065060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.065003    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:30:03.065568 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.065556    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 00:30:03.065640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.065570    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:30:03.065640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.065568    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 00:30:03.065705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.065657    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 00:30:03.066194 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.066177    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:30:03.066231 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.066209    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:30:03.066231 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.066219    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:30:03.067972 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.067958    2576 policy_none.go:49] "None policy: Start"
Sep 17 00:30:03.068028 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.067978    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 00:30:03.068028 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.067991    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 00:30:03.069972 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.069958    2576 policy_none.go:47] "Start"
Sep 17 00:30:03.104800 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.104786    2576 manager.go:337] "Starting Device Plugin manager"
Sep 17 00:30:03.123092 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.104825    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 00:30:03.123092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.104840    2576 server.go:87] "Starting device plugin registration server"
Sep 17 00:30:03.123092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.105167    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 00:30:03.123092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.105179    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 00:30:03.123092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.105293    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 00:30:03.123092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.105360    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 00:30:03.123092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.105369    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 00:30:03.123092 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.105835    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 00:30:03.123092 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.105874    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.176687 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.176648    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 00:30:03.178032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.178015    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 00:30:03.178097 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.178071    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 00:30:03.178097 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.178094    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 00:30:03.178163 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.178141    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 00:30:03.180076 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.180060    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:30:03.206345 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.206272    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:30:03.207118 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.207104    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:30:03.207172 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.207134    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:30:03.207172 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.207150    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:30:03.207230 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.207178    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.214862 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.214847    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.214908 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.214867    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-51.ec2.internal\": node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.224572 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.224554    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.279069 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.279045    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal"]
Sep 17 00:30:03.279161 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.279118    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:30:03.280287 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.280272    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:30:03.280353 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.280304    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:30:03.280353 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.280318    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:30:03.282800 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.282782    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:30:03.282921 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.282902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.283004 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.282956    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:30:03.284744 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.284725    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:30:03.284835 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.284728    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:30:03.284835 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.284800    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:30:03.284835 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.284759    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:30:03.284943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.284837    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:30:03.284943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.284817    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:30:03.287506 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.287488    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.287589 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.287523    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:30:03.288205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.288189    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:30:03.288277 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.288215    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:30:03.288277 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.288227    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:30:03.309807 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.309785    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-51.ec2.internal\" not found" node="ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.314174 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.314159    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-51.ec2.internal\" not found" node="ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.324636 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.324613    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.346643 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.346623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/959cb07438d5c8900b2bb472409b47d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal\" (UID: \"959cb07438d5c8900b2bb472409b47d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.346698 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.346648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/959cb07438d5c8900b2bb472409b47d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal\" (UID: \"959cb07438d5c8900b2bb472409b47d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.346698 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.346662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b66c31ea21941fb1ad957ccbcb0a0288-config\") pod \"kube-apiserver-proxy-ip-10-0-143-51.ec2.internal\" (UID: \"b66c31ea21941fb1ad957ccbcb0a0288\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.425284 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.425257    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.447673 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.447647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b66c31ea21941fb1ad957ccbcb0a0288-config\") pod \"kube-apiserver-proxy-ip-10-0-143-51.ec2.internal\" (UID: \"b66c31ea21941fb1ad957ccbcb0a0288\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.447747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.447684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/959cb07438d5c8900b2bb472409b47d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal\" (UID: \"959cb07438d5c8900b2bb472409b47d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.447747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.447704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/959cb07438d5c8900b2bb472409b47d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal\" (UID: \"959cb07438d5c8900b2bb472409b47d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.447815 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.447762    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/959cb07438d5c8900b2bb472409b47d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal\" (UID: \"959cb07438d5c8900b2bb472409b47d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.447815 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.447770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b66c31ea21941fb1ad957ccbcb0a0288-config\") pod \"kube-apiserver-proxy-ip-10-0-143-51.ec2.internal\" (UID: \"b66c31ea21941fb1ad957ccbcb0a0288\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.447815 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.447770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/959cb07438d5c8900b2bb472409b47d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal\" (UID: \"959cb07438d5c8900b2bb472409b47d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.526091 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.526024    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.612587 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.612551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.616447 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.616430    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:03.626126 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.626106    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.726672 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.726634    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.827194 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.827121    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.927804 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:03.927778    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:03.973246 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.973219    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 00:30:03.973640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.973368    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 00:30:03.973640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:03.973374    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 00:30:04.028660 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:04.028631    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:04.040821 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:04.040791    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 00:25:04 +0000 UTC" deadline="2028-03-15 17:34:04.312424369 +0000 UTC"
Sep 17 00:30:04.040821 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:04.040817    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13097h4m0.271609839s"
Sep 17 00:30:04.129129 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:04.129108    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:04.229709 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:04.229659    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:04.277483 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:04.277454    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66c31ea21941fb1ad957ccbcb0a0288.slice/crio-4120240f80b76daa50bd61fdd1f91413ad6ca22537dc629d9b9c0e118dd0fb8a WatchSource:0}: Error finding container 4120240f80b76daa50bd61fdd1f91413ad6ca22537dc629d9b9c0e118dd0fb8a: Status 404 returned error can't find the container with id 4120240f80b76daa50bd61fdd1f91413ad6ca22537dc629d9b9c0e118dd0fb8a
Sep 17 00:30:04.277702 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:04.277680    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959cb07438d5c8900b2bb472409b47d9.slice/crio-ebd43dbc94b774eedf52213ba4fb4c48782eef6c0fcae594350dd46dcf601af3 WatchSource:0}: Error finding container ebd43dbc94b774eedf52213ba4fb4c48782eef6c0fcae594350dd46dcf601af3: Status 404 returned error can't find the container with id ebd43dbc94b774eedf52213ba4fb4c48782eef6c0fcae594350dd46dcf601af3
Sep 17 00:30:04.282583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:04.282569    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:30:04.330808 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:04.330781    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:04.388479 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:04.388413    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:30:04.431213 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:04.431167    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:05.934934 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:05.934898    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-51.ec2.internal\" not found"
Sep 17 00:30:05.970369 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:05.970343    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:30:06.047494 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.047421    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:06.058034 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.058017    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 00:30:06.058995 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.058976    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal"
Sep 17 00:30:06.066656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.066637    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 00:30:06.424037 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.423958    2576 apiserver.go:52] "Watching apiserver"
Sep 17 00:30:06.431188 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.431162    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 00:30:06.431984 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.431956    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mlq9c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66","openshift-cluster-node-tuning-operator/tuned-6v7j7","openshift-image-registry/node-ca-cf4c4","openshift-multus/multus-additional-cni-plugins-84cm9","openshift-network-diagnostics/network-check-target-sdhcm","openshift-ovn-kubernetes/ovnkube-node-8tfff","kube-system/konnectivity-agent-4wr6h","kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal","openshift-dns/node-resolver-vhmzk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal","openshift-multus/multus-rdtmd","openshift-multus/network-metrics-daemon-w9nj2","openshift-network-operator/iptables-alerter-rv25s"]
Sep 17 00:30:06.435940 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.435914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.438515 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.438492    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 00:30:06.439188 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.439169    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 00:30:06.439712 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.439693    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 00:30:06.439787 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.439749    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 00:30:06.439885 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.439872    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 00:30:06.439960 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.439944    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-47w5m\""
Sep 17 00:30:06.440035 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.440020    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 00:30:06.440222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.440205    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:06.440302 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.440274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:06.441500 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.440953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.443043 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.443023    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.443467 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.443443    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wh4r5\""
Sep 17 00:30:06.443562 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.443532    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 00:30:06.443562 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.443545    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 00:30:06.444952 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.444932    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 00:30:06.445220 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.445196    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 00:30:06.445424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.445405    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqndc\""
Sep 17 00:30:06.445726 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.445707    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 00:30:06.446347 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.446327    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 00:30:06.447416 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.447395    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.449715 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.449696    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 00:30:06.449863 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.449849    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 00:30:06.449997 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.449984    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 00:30:06.449997 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.449991    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lvf5v\""
Sep 17 00:30:06.450201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.450185    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 00:30:06.450201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.450209    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 00:30:06.450361 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.450354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:06.450440 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.450416    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:06.452634 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.452462    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.455097 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.455077    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 00:30:06.455608 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.455304    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:06.455608 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.455386    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:06.455741 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.455728    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 00:30:06.455940 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.455920    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 00:30:06.456127 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.456110    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tmsdl\""
Sep 17 00:30:06.458514 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458495    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.458777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.458884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.458884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kvl7\" (UniqueName: \"kubernetes.io/projected/1787d2e6-4f22-41d1-bee6-d87f15c5936d-kube-api-access-4kvl7\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.458884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-systemd-units\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.458884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-slash\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-run-systemd\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-cni-bin\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458935    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/116ae241-a23d-4780-ae66-a3ba9aee4499-ovnkube-config\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.458989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-host\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.459084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-tuned\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.459084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-modprobe-d\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.459084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-kubelet\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-etc-openvswitch\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/116ae241-a23d-4780-ae66-a3ba9aee4499-ovnkube-script-lib\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459135    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmqp\" (UniqueName: \"kubernetes.io/projected/116ae241-a23d-4780-ae66-a3ba9aee4499-kube-api-access-8vmqp\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459174    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8p6c\" (UniqueName: \"kubernetes.io/projected/06683d51-91e1-4e99-a67d-75d8a715b5ab-kube-api-access-l8p6c\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-sysctl-conf\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42d25648-164b-4ad8-9d6e-d38f1f3301e4-host\") pod \"node-ca-cf4c4\" (UID: \"42d25648-164b-4ad8-9d6e-d38f1f3301e4\") " pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-node-log\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-log-socket\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-kubernetes\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-sys\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-var-lib-kubelet\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42d25648-164b-4ad8-9d6e-d38f1f3301e4-serviceca\") pod \"node-ca-cf4c4\" (UID: \"42d25648-164b-4ad8-9d6e-d38f1f3301e4\") " pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-cnibin\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-run-openvswitch\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459390    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/116ae241-a23d-4780-ae66-a3ba9aee4499-ovn-node-metrics-cert\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.459424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-run\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk6ks\" (UniqueName: \"kubernetes.io/projected/42d25648-164b-4ad8-9d6e-d38f1f3301e4-kube-api-access-pk6ks\") pod \"node-ca-cf4c4\" (UID: \"42d25648-164b-4ad8-9d6e-d38f1f3301e4\") " pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459484    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-system-cni-dir\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-var-lib-openvswitch\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-cni-netd\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1787d2e6-4f22-41d1-bee6-d87f15c5936d-tmp\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jwd\" (UniqueName: \"kubernetes.io/projected/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-kube-api-access-97jwd\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-sysconfig\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/116ae241-a23d-4780-ae66-a3ba9aee4499-env-overrides\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-sysctl-d\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-run-netns\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-run-ovn\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.460222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-systemd\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.460855 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-lib-modules\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.460855 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.460855 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.459959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.461003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.460885    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m68w8\""
Sep 17 00:30:06.461003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.460981    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 00:30:06.461110 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.460886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.461286 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.461201    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 00:30:06.461487 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.461457    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 00:30:06.465276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.463674    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 00:30:06.465276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.463877    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n4jsj\""
Sep 17 00:30:06.465276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.464062    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 00:30:06.465276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.464452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:06.465276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.464820    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:30:06.467387 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.467364    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 00:30:06.467705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.467520    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 00:30:06.467705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.467644    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kt69v\""
Sep 17 00:30:06.468453 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.468419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.470991 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.470802    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 00:30:06.470991 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.470828    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bdws5\""
Sep 17 00:30:06.476917 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.476894    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gjsvp"
Sep 17 00:30:06.483728 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.483390    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gjsvp"
Sep 17 00:30:06.549112 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.549079    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 00:30:06.560450 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.560583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kvl7\" (UniqueName: \"kubernetes.io/projected/1787d2e6-4f22-41d1-bee6-d87f15c5936d-kube-api-access-4kvl7\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.560583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-etc-kubernetes\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.560583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.560583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560521    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.560583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560542    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:06.560583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-cni-dir\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-run-k8s-cni-cncf-io\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-systemd-units\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/116ae241-a23d-4780-ae66-a3ba9aee4499-ovnkube-config\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-run-netns\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jwd\" (UniqueName: \"kubernetes.io/projected/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-kube-api-access-97jwd\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-kubelet\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-etc-openvswitch\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/116ae241-a23d-4780-ae66-a3ba9aee4499-ovnkube-script-lib\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmqp\" (UniqueName: \"kubernetes.io/projected/116ae241-a23d-4780-ae66-a3ba9aee4499-kube-api-access-8vmqp\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8p6c\" (UniqueName: \"kubernetes.io/projected/06683d51-91e1-4e99-a67d-75d8a715b5ab-kube-api-access-l8p6c\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42d25648-164b-4ad8-9d6e-d38f1f3301e4-host\") pod \"node-ca-cf4c4\" (UID: \"42d25648-164b-4ad8-9d6e-d38f1f3301e4\") " pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.560890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-var-lib-kubelet\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-node-log\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-log-socket\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-var-lib-kubelet\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-hostroot\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-daemon-config\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-run-multus-certs\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bm2\" (UniqueName: \"kubernetes.io/projected/3774f212-bf63-418d-86af-17d31c4adcc9-kube-api-access-r2bm2\") pod \"iptables-alerter-rv25s\" (UID: \"3774f212-bf63-418d-86af-17d31c4adcc9\") " pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-sys-fs\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/116ae241-a23d-4780-ae66-a3ba9aee4499-ovn-node-metrics-cert\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6ks\" (UniqueName: \"kubernetes.io/projected/42d25648-164b-4ad8-9d6e-d38f1f3301e4-kube-api-access-pk6ks\") pod \"node-ca-cf4c4\" (UID: \"42d25648-164b-4ad8-9d6e-d38f1f3301e4\") " pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-conf-dir\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.560760    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-systemd-units\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/289b9bc3-1131-473a-a889-0cdb22286663-kubelet-config\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:06.561548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561322    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42d25648-164b-4ad8-9d6e-d38f1f3301e4-host\") pod \"node-ca-cf4c4\" (UID: \"42d25648-164b-4ad8-9d6e-d38f1f3301e4\") " pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561712    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-node-log\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561749    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-var-lib-kubelet\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561928    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-kubelet\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561992    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-log-socket\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59884e4c-8c0b-4b73-9c4a-b9d0490ad534-agent-certs\") pod \"konnectivity-agent-4wr6h\" (UID: \"59884e4c-8c0b-4b73-9c4a-b9d0490ad534\") " pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-var-lib-openvswitch\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-cni-netd\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1787d2e6-4f22-41d1-bee6-d87f15c5936d-tmp\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/116ae241-a23d-4780-ae66-a3ba9aee4499-ovnkube-script-lib\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562176    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-etc-openvswitch\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-var-lib-cni-multus\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562217    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-cni-netd\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-var-lib-openvswitch\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99ec850-d4b6-4541-90a4-d158112c2570-tmp-dir\") pod \"node-resolver-vhmzk\" (UID: \"c99ec850-d4b6-4541-90a4-d158112c2570\") " pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/289b9bc3-1131-473a-a889-0cdb22286663-dbus\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:06.562320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c99ec850-d4b6-4541-90a4-d158112c2570-hosts-file\") pod \"node-resolver-vhmzk\" (UID: \"c99ec850-d4b6-4541-90a4-d158112c2570\") " pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.561791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/116ae241-a23d-4780-ae66-a3ba9aee4499-ovnkube-config\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562365    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-run-ovn\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-run-ovn\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-systemd\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-systemd\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562489    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-system-cni-dir\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3774f212-bf63-418d-86af-17d31c4adcc9-host-slash\") pod \"iptables-alerter-rv25s\" (UID: \"3774f212-bf63-418d-86af-17d31c4adcc9\") " pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3774f212-bf63-418d-86af-17d31c4adcc9-iptables-alerter-script\") pod \"iptables-alerter-rv25s\" (UID: \"3774f212-bf63-418d-86af-17d31c4adcc9\") " pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-slash\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-run-systemd\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562739    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-cni-bin\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.563228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-host\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-tuned\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562849    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-var-lib-cni-bin\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562911    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-slash\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562943    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-cni-bin\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.562977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-run-systemd\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563011    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-device-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-modprobe-d\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563117    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-host\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563124    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-sysctl-conf\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-kubernetes\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-sys\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-modprobe-d\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42d25648-164b-4ad8-9d6e-d38f1f3301e4-serviceca\") pod \"node-ca-cf4c4\" (UID: \"42d25648-164b-4ad8-9d6e-d38f1f3301e4\") " pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.564027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-cnibin\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-socket-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563335    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbx8z\" (UniqueName: \"kubernetes.io/projected/c99ec850-d4b6-4541-90a4-d158112c2570-kube-api-access-vbx8z\") pod \"node-resolver-vhmzk\" (UID: \"c99ec850-d4b6-4541-90a4-d158112c2570\") " pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563336    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-sysctl-conf\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59884e4c-8c0b-4b73-9c4a-b9d0490ad534-konnectivity-ca\") pod \"konnectivity-agent-4wr6h\" (UID: \"59884e4c-8c0b-4b73-9c4a-b9d0490ad534\") " pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-run-openvswitch\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-run\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-kubernetes\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-system-cni-dir\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563468    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-cnibin\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563476    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563507    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-run-openvswitch\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-registration-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgtk\" (UniqueName: \"kubernetes.io/projected/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-kube-api-access-wtgtk\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-cnibin\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563616    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-sys\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28bc9500-7f38-4e77-98c7-86001cac7fd9-cni-binary-copy\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.564829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-socket-dir-parent\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-sysconfig\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563704    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-system-cni-dir\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/116ae241-a23d-4780-ae66-a3ba9aee4499-env-overrides\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563741    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42d25648-164b-4ad8-9d6e-d38f1f3301e4-serviceca\") pod \"node-ca-cf4c4\" (UID: \"42d25648-164b-4ad8-9d6e-d38f1f3301e4\") " pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-sysconfig\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563823    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-run\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.563888    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.563925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-sysctl-d\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.563988    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs podName:06683d51-91e1-4e99-a67d-75d8a715b5ab nodeName:}" failed. No retries permitted until 2026-09-17 00:30:07.063952943 +0000 UTC m=+3.019549278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs") pod "network-metrics-daemon-w9nj2" (UID: "06683d51-91e1-4e99-a67d-75d8a715b5ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.564011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.564032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-sysctl-d\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.564067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-run-netns\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.564094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-lib-modules\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.564133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmcj\" (UniqueName: \"kubernetes.io/projected/28bc9500-7f38-4e77-98c7-86001cac7fd9-kube-api-access-rnmcj\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.564258    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/116ae241-a23d-4780-ae66-a3ba9aee4499-host-run-netns\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.565621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.564263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1787d2e6-4f22-41d1-bee6-d87f15c5936d-lib-modules\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.569862 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.564276    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/116ae241-a23d-4780-ae66-a3ba9aee4499-env-overrides\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.569862 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.566611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1787d2e6-4f22-41d1-bee6-d87f15c5936d-etc-tuned\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.569862 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.566861    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/116ae241-a23d-4780-ae66-a3ba9aee4499-ovn-node-metrics-cert\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.569862 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.567422    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1787d2e6-4f22-41d1-bee6-d87f15c5936d-tmp\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.570944 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.570657    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:30:06.570944 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.570681    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:30:06.570944 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.570694    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zfvd5 for pod openshift-network-diagnostics/network-check-target-sdhcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:06.570944 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.570772    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5 podName:009bd9cc-b8a9-4c68-9a8e-158b348866cb nodeName:}" failed. No retries permitted until 2026-09-17 00:30:07.070744545 +0000 UTC m=+3.026340881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zfvd5" (UniqueName: "kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5") pod "network-check-target-sdhcm" (UID: "009bd9cc-b8a9-4c68-9a8e-158b348866cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:06.574171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.574067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kvl7\" (UniqueName: \"kubernetes.io/projected/1787d2e6-4f22-41d1-bee6-d87f15c5936d-kube-api-access-4kvl7\") pod \"tuned-6v7j7\" (UID: \"1787d2e6-4f22-41d1-bee6-d87f15c5936d\") " pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.574171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.574161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmqp\" (UniqueName: \"kubernetes.io/projected/116ae241-a23d-4780-ae66-a3ba9aee4499-kube-api-access-8vmqp\") pod \"ovnkube-node-8tfff\" (UID: \"116ae241-a23d-4780-ae66-a3ba9aee4499\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.574572 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.574553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jwd\" (UniqueName: \"kubernetes.io/projected/50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8-kube-api-access-97jwd\") pod \"multus-additional-cni-plugins-84cm9\" (UID: \"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8\") " pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.574772 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.574752    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8p6c\" (UniqueName: \"kubernetes.io/projected/06683d51-91e1-4e99-a67d-75d8a715b5ab-kube-api-access-l8p6c\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:06.576792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.576767    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6ks\" (UniqueName: \"kubernetes.io/projected/42d25648-164b-4ad8-9d6e-d38f1f3301e4-kube-api-access-pk6ks\") pod \"node-ca-cf4c4\" (UID: \"42d25648-164b-4ad8-9d6e-d38f1f3301e4\") " pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.587033 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.586997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal" event={"ID":"959cb07438d5c8900b2bb472409b47d9","Type":"ContainerStarted","Data":"ebd43dbc94b774eedf52213ba4fb4c48782eef6c0fcae594350dd46dcf601af3"}
Sep 17 00:30:06.588648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.588612    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal" event={"ID":"b66c31ea21941fb1ad957ccbcb0a0288","Type":"ContainerStarted","Data":"4120240f80b76daa50bd61fdd1f91413ad6ca22537dc629d9b9c0e118dd0fb8a"}
Sep 17 00:30:06.664513 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-var-lib-cni-bin\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.664513 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-device-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.664732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-socket-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.664732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbx8z\" (UniqueName: \"kubernetes.io/projected/c99ec850-d4b6-4541-90a4-d158112c2570-kube-api-access-vbx8z\") pod \"node-resolver-vhmzk\" (UID: \"c99ec850-d4b6-4541-90a4-d158112c2570\") " pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.664732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59884e4c-8c0b-4b73-9c4a-b9d0490ad534-konnectivity-ca\") pod \"konnectivity-agent-4wr6h\" (UID: \"59884e4c-8c0b-4b73-9c4a-b9d0490ad534\") " pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:06.664732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.664732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-registration-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.664732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgtk\" (UniqueName: \"kubernetes.io/projected/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-kube-api-access-wtgtk\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.664732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-cnibin\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.664732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664702    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28bc9500-7f38-4e77-98c7-86001cac7fd9-cni-binary-copy\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.664732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-socket-dir-parent\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmcj\" (UniqueName: \"kubernetes.io/projected/28bc9500-7f38-4e77-98c7-86001cac7fd9-kube-api-access-rnmcj\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-etc-kubernetes\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-cni-dir\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-run-k8s-cni-cncf-io\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-run-netns\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-var-lib-kubelet\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-hostroot\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.664982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-daemon-config\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-run-multus-certs\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bm2\" (UniqueName: \"kubernetes.io/projected/3774f212-bf63-418d-86af-17d31c4adcc9-kube-api-access-r2bm2\") pod \"iptables-alerter-rv25s\" (UID: \"3774f212-bf63-418d-86af-17d31c4adcc9\") " pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-sys-fs\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.665093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-conf-dir\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/289b9bc3-1131-473a-a889-0cdb22286663-kubelet-config\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59884e4c-8c0b-4b73-9c4a-b9d0490ad534-agent-certs\") pod \"konnectivity-agent-4wr6h\" (UID: \"59884e4c-8c0b-4b73-9c4a-b9d0490ad534\") " pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-var-lib-cni-multus\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99ec850-d4b6-4541-90a4-d158112c2570-tmp-dir\") pod \"node-resolver-vhmzk\" (UID: \"c99ec850-d4b6-4541-90a4-d158112c2570\") " pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/289b9bc3-1131-473a-a889-0cdb22286663-dbus\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c99ec850-d4b6-4541-90a4-d158112c2570-hosts-file\") pod \"node-resolver-vhmzk\" (UID: \"c99ec850-d4b6-4541-90a4-d158112c2570\") " pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-system-cni-dir\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3774f212-bf63-418d-86af-17d31c4adcc9-host-slash\") pod \"iptables-alerter-rv25s\" (UID: \"3774f212-bf63-418d-86af-17d31c4adcc9\") " pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.665714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3774f212-bf63-418d-86af-17d31c4adcc9-iptables-alerter-script\") pod \"iptables-alerter-rv25s\" (UID: \"3774f212-bf63-418d-86af-17d31c4adcc9\") " pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.666205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665928    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3774f212-bf63-418d-86af-17d31c4adcc9-iptables-alerter-script\") pod \"iptables-alerter-rv25s\" (UID: \"3774f212-bf63-418d-86af-17d31c4adcc9\") " pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.666205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.665995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-var-lib-cni-bin\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.666205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.666045    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-device-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.666205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.666132    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-socket-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.666897 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.666873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59884e4c-8c0b-4b73-9c4a-b9d0490ad534-konnectivity-ca\") pod \"konnectivity-agent-4wr6h\" (UID: \"59884e4c-8c0b-4b73-9c4a-b9d0490ad534\") " pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:06.666983 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.666953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.667034 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.667006    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-registration-dir\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.667285 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.667262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-cnibin\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.667756 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.667736    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28bc9500-7f38-4e77-98c7-86001cac7fd9-cni-binary-copy\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.667838 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.667809    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-socket-dir-parent\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.667931 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.667913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.668254 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.668201    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-etc-kubernetes\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.668349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.668274    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-cni-dir\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.668349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.668317    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-run-k8s-cni-cncf-io\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.668411 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.668360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-run-netns\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.668411 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.668404    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-var-lib-kubelet\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.668483 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.668446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-hostroot\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.668883 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.668858    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-daemon-config\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.668937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.668918    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-run-multus-certs\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.669239 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.669215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-host-var-lib-cni-multus\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.669316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.669244    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-multus-conf-dir\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.669316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.669278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/289b9bc3-1131-473a-a889-0cdb22286663-kubelet-config\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:06.669316 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.669313    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:06.669492 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:06.669368    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret podName:289b9bc3-1131-473a-a889-0cdb22286663 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:07.169351361 +0000 UTC m=+3.124947703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret") pod "global-pull-secret-syncer-mlq9c" (UID: "289b9bc3-1131-473a-a889-0cdb22286663") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:06.669492 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.669370    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c99ec850-d4b6-4541-90a4-d158112c2570-hosts-file\") pod \"node-resolver-vhmzk\" (UID: \"c99ec850-d4b6-4541-90a4-d158112c2570\") " pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.669603 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.669573    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28bc9500-7f38-4e77-98c7-86001cac7fd9-system-cni-dir\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.669638 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.669606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-sys-fs\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.669638 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.669605    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/289b9bc3-1131-473a-a889-0cdb22286663-dbus\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:06.669714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.669636    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3774f212-bf63-418d-86af-17d31c4adcc9-host-slash\") pod \"iptables-alerter-rv25s\" (UID: \"3774f212-bf63-418d-86af-17d31c4adcc9\") " pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.669714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.669683    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99ec850-d4b6-4541-90a4-d158112c2570-tmp-dir\") pod \"node-resolver-vhmzk\" (UID: \"c99ec850-d4b6-4541-90a4-d158112c2570\") " pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.672231 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.672163    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59884e4c-8c0b-4b73-9c4a-b9d0490ad534-agent-certs\") pod \"konnectivity-agent-4wr6h\" (UID: \"59884e4c-8c0b-4b73-9c4a-b9d0490ad534\") " pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:06.675544 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.675482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmcj\" (UniqueName: \"kubernetes.io/projected/28bc9500-7f38-4e77-98c7-86001cac7fd9-kube-api-access-rnmcj\") pod \"multus-rdtmd\" (UID: \"28bc9500-7f38-4e77-98c7-86001cac7fd9\") " pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:06.675878 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.675850    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgtk\" (UniqueName: \"kubernetes.io/projected/4a4a2e00-6841-4a9b-ab3d-f86001cf95f5-kube-api-access-wtgtk\") pod \"aws-ebs-csi-driver-node-5cd66\" (UID: \"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.675977 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.675852    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbx8z\" (UniqueName: \"kubernetes.io/projected/c99ec850-d4b6-4541-90a4-d158112c2570-kube-api-access-vbx8z\") pod \"node-resolver-vhmzk\" (UID: \"c99ec850-d4b6-4541-90a4-d158112c2570\") " pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.676866 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.676845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bm2\" (UniqueName: \"kubernetes.io/projected/3774f212-bf63-418d-86af-17d31c4adcc9-kube-api-access-r2bm2\") pod \"iptables-alerter-rv25s\" (UID: \"3774f212-bf63-418d-86af-17d31c4adcc9\") " pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.719790 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.719755    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:30:06.751331 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.751302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:06.757888 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.757866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6v7j7"
Sep 17 00:30:06.771726 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.771703    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cf4c4"
Sep 17 00:30:06.778349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.778324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84cm9"
Sep 17 00:30:06.789866 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.789847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rv25s"
Sep 17 00:30:06.810455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.810435    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66"
Sep 17 00:30:06.817072 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.817052    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vhmzk"
Sep 17 00:30:06.823784 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.823767    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:06.829313 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:06.829298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rdtmd"
Sep 17 00:30:07.067893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.067811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:07.068053 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:07.067976    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:07.068053 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:07.068050    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs podName:06683d51-91e1-4e99-a67d-75d8a715b5ab nodeName:}" failed. No retries permitted until 2026-09-17 00:30:08.068030805 +0000 UTC m=+4.023627137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs") pod "network-metrics-daemon-w9nj2" (UID: "06683d51-91e1-4e99-a67d-75d8a715b5ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:07.168567 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.168537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:07.168714 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:07.168695    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:30:07.168748 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:07.168714    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:30:07.168748 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:07.168723    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zfvd5 for pod openshift-network-diagnostics/network-check-target-sdhcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:07.168805 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:07.168773    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5 podName:009bd9cc-b8a9-4c68-9a8e-158b348866cb nodeName:}" failed. No retries permitted until 2026-09-17 00:30:08.168758224 +0000 UTC m=+4.124354547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfvd5" (UniqueName: "kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5") pod "network-check-target-sdhcm" (UID: "009bd9cc-b8a9-4c68-9a8e-158b348866cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:07.269597 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.269566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:07.269756 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:07.269683    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:07.269756 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:07.269732    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret podName:289b9bc3-1131-473a-a889-0cdb22286663 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:08.269718767 +0000 UTC m=+4.225315086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret") pod "global-pull-secret-syncer-mlq9c" (UID: "289b9bc3-1131-473a-a889-0cdb22286663") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:07.486816 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.486748    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 00:25:06 +0000 UTC" deadline="2028-04-29 21:32:41.60971557 +0000 UTC"
Sep 17 00:30:07.486816 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.486778    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14181h2m34.12294125s"
Sep 17 00:30:07.546229 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:07.546090    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d25648_164b_4ad8_9d6e_d38f1f3301e4.slice/crio-689f0aef10596da02695bd790f6aa225552e2b035d4f6f0686971cd560e6feeb WatchSource:0}: Error finding container 689f0aef10596da02695bd790f6aa225552e2b035d4f6f0686971cd560e6feeb: Status 404 returned error can't find the container with id 689f0aef10596da02695bd790f6aa225552e2b035d4f6f0686971cd560e6feeb
Sep 17 00:30:07.547994 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:07.547970    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1787d2e6_4f22_41d1_bee6_d87f15c5936d.slice/crio-81a17732f3afe68f257c234f069cc62ba5873ad5ea9238ac9f905b11da8dc044 WatchSource:0}: Error finding container 81a17732f3afe68f257c234f069cc62ba5873ad5ea9238ac9f905b11da8dc044: Status 404 returned error can't find the container with id 81a17732f3afe68f257c234f069cc62ba5873ad5ea9238ac9f905b11da8dc044
Sep 17 00:30:07.548847 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:07.548778    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99ec850_d4b6_4541_90a4_d158112c2570.slice/crio-a18fe5a61be935193e48e6fe5f4ec8b0934f23faf8193697a6eff5aa96526cc8 WatchSource:0}: Error finding container a18fe5a61be935193e48e6fe5f4ec8b0934f23faf8193697a6eff5aa96526cc8: Status 404 returned error can't find the container with id a18fe5a61be935193e48e6fe5f4ec8b0934f23faf8193697a6eff5aa96526cc8
Sep 17 00:30:07.550861 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:07.550401    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bc9500_7f38_4e77_98c7_86001cac7fd9.slice/crio-1c827ad9094cea235cffa18474d409cae5a830c87fcf524e075b5a5fe2d80801 WatchSource:0}: Error finding container 1c827ad9094cea235cffa18474d409cae5a830c87fcf524e075b5a5fe2d80801: Status 404 returned error can't find the container with id 1c827ad9094cea235cffa18474d409cae5a830c87fcf524e075b5a5fe2d80801
Sep 17 00:30:07.552822 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:07.552404    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116ae241_a23d_4780_ae66_a3ba9aee4499.slice/crio-acf837c5886aed8d2326f831bf8dc34c8293840ffa6d751f7c63fd18ac057704 WatchSource:0}: Error finding container acf837c5886aed8d2326f831bf8dc34c8293840ffa6d751f7c63fd18ac057704: Status 404 returned error can't find the container with id acf837c5886aed8d2326f831bf8dc34c8293840ffa6d751f7c63fd18ac057704
Sep 17 00:30:07.554762 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:07.554382    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4a2e00_6841_4a9b_ab3d_f86001cf95f5.slice/crio-ca9175dd21f639c0ae491a55e6e0747bdad8e6b732db9e8d50403cd25b142242 WatchSource:0}: Error finding container ca9175dd21f639c0ae491a55e6e0747bdad8e6b732db9e8d50403cd25b142242: Status 404 returned error can't find the container with id ca9175dd21f639c0ae491a55e6e0747bdad8e6b732db9e8d50403cd25b142242
Sep 17 00:30:07.554762 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:07.554688    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59884e4c_8c0b_4b73_9c4a_b9d0490ad534.slice/crio-c61919f4b97aaac3cc81103f1d1e97cd2cc87b0a017c6fdcdc255a79fbb68b83 WatchSource:0}: Error finding container c61919f4b97aaac3cc81103f1d1e97cd2cc87b0a017c6fdcdc255a79fbb68b83: Status 404 returned error can't find the container with id c61919f4b97aaac3cc81103f1d1e97cd2cc87b0a017c6fdcdc255a79fbb68b83
Sep 17 00:30:07.557428 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:07.557013    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c88f96_54de_4bbc_a2f1_7b0fb6ff18e8.slice/crio-aaf154ac44cef33cdad1708f422afffa1a0fdd29c034cd55f5c43dbba46a5a2c WatchSource:0}: Error finding container aaf154ac44cef33cdad1708f422afffa1a0fdd29c034cd55f5c43dbba46a5a2c: Status 404 returned error can't find the container with id aaf154ac44cef33cdad1708f422afffa1a0fdd29c034cd55f5c43dbba46a5a2c
Sep 17 00:30:07.557663 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:07.557647    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3774f212_bf63_418d_86af_17d31c4adcc9.slice/crio-aa3e9235b2cbc3b388cd94da71b12dda454fe93f3602541b6adc606fbd1f11c1 WatchSource:0}: Error finding container aa3e9235b2cbc3b388cd94da71b12dda454fe93f3602541b6adc606fbd1f11c1: Status 404 returned error can't find the container with id aa3e9235b2cbc3b388cd94da71b12dda454fe93f3602541b6adc606fbd1f11c1
Sep 17 00:30:07.581430 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.581409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:07.581531 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:07.581506    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:07.593069 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.593042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rdtmd" event={"ID":"28bc9500-7f38-4e77-98c7-86001cac7fd9","Type":"ContainerStarted","Data":"1c827ad9094cea235cffa18474d409cae5a830c87fcf524e075b5a5fe2d80801"}
Sep 17 00:30:07.594110 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.594082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cf4c4" event={"ID":"42d25648-164b-4ad8-9d6e-d38f1f3301e4","Type":"ContainerStarted","Data":"689f0aef10596da02695bd790f6aa225552e2b035d4f6f0686971cd560e6feeb"}
Sep 17 00:30:07.595499 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.595469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vhmzk" event={"ID":"c99ec850-d4b6-4541-90a4-d158112c2570","Type":"ContainerStarted","Data":"a18fe5a61be935193e48e6fe5f4ec8b0934f23faf8193697a6eff5aa96526cc8"}
Sep 17 00:30:07.596494 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.596474    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6v7j7" event={"ID":"1787d2e6-4f22-41d1-bee6-d87f15c5936d","Type":"ContainerStarted","Data":"81a17732f3afe68f257c234f069cc62ba5873ad5ea9238ac9f905b11da8dc044"}
Sep 17 00:30:07.599491 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.599472    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rv25s" event={"ID":"3774f212-bf63-418d-86af-17d31c4adcc9","Type":"ContainerStarted","Data":"aa3e9235b2cbc3b388cd94da71b12dda454fe93f3602541b6adc606fbd1f11c1"}
Sep 17 00:30:07.602511 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.602484    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wr6h" event={"ID":"59884e4c-8c0b-4b73-9c4a-b9d0490ad534","Type":"ContainerStarted","Data":"c61919f4b97aaac3cc81103f1d1e97cd2cc87b0a017c6fdcdc255a79fbb68b83"}
Sep 17 00:30:07.603497 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.603472    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerStarted","Data":"aaf154ac44cef33cdad1708f422afffa1a0fdd29c034cd55f5c43dbba46a5a2c"}
Sep 17 00:30:07.605050 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.604720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66" event={"ID":"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5","Type":"ContainerStarted","Data":"ca9175dd21f639c0ae491a55e6e0747bdad8e6b732db9e8d50403cd25b142242"}
Sep 17 00:30:07.605899 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:07.605881    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerStarted","Data":"acf837c5886aed8d2326f831bf8dc34c8293840ffa6d751f7c63fd18ac057704"}
Sep 17 00:30:08.074851 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.074774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:08.075107 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.074902    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:08.075107 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.074966    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs podName:06683d51-91e1-4e99-a67d-75d8a715b5ab nodeName:}" failed. No retries permitted until 2026-09-17 00:30:10.074947675 +0000 UTC m=+6.030544017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs") pod "network-metrics-daemon-w9nj2" (UID: "06683d51-91e1-4e99-a67d-75d8a715b5ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:08.184163 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.179684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:08.184163 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.179941    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:30:08.184163 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.179961    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:30:08.184163 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.179976    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zfvd5 for pod openshift-network-diagnostics/network-check-target-sdhcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:08.184163 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.180041    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5 podName:009bd9cc-b8a9-4c68-9a8e-158b348866cb nodeName:}" failed. No retries permitted until 2026-09-17 00:30:10.180022363 +0000 UTC m=+6.135618687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfvd5" (UniqueName: "kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5") pod "network-check-target-sdhcm" (UID: "009bd9cc-b8a9-4c68-9a8e-158b348866cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:08.281686 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.281108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:08.281686 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.281286    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:08.281686 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.281344    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret podName:289b9bc3-1131-473a-a889-0cdb22286663 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:10.281326856 +0000 UTC m=+6.236923175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret") pod "global-pull-secret-syncer-mlq9c" (UID: "289b9bc3-1131-473a-a889-0cdb22286663") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:08.487888 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.487754    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 00:25:06 +0000 UTC" deadline="2028-04-04 09:39:09.029781195 +0000 UTC"
Sep 17 00:30:08.487888 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.487810    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13569h9m0.541974908s"
Sep 17 00:30:08.582777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.582281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:08.582777 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.582419    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:08.583257 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.583095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:08.583257 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:08.583215    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:08.620750 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.619782    2576 generic.go:356] "Generic (PLEG): container finished" podID="959cb07438d5c8900b2bb472409b47d9" containerID="38715fd848b390ac17171ad4a0a34974f40595073d0014b06f9ce8a7c27bfc8e" exitCode=0
Sep 17 00:30:08.620750 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.620481    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal" event={"ID":"959cb07438d5c8900b2bb472409b47d9","Type":"ContainerDied","Data":"38715fd848b390ac17171ad4a0a34974f40595073d0014b06f9ce8a7c27bfc8e"}
Sep 17 00:30:08.632820 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.632786    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal" event={"ID":"b66c31ea21941fb1ad957ccbcb0a0288","Type":"ContainerStarted","Data":"58761605d0280a48f0a13be074a6a36611411e7c338efd7d8c4c30ebfb34ec81"}
Sep 17 00:30:08.649705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:08.649652    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-51.ec2.internal" podStartSLOduration=2.649632871 podStartE2EDuration="2.649632871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:08.649465618 +0000 UTC m=+4.605061961" watchObservedRunningTime="2026-09-17 00:30:08.649632871 +0000 UTC m=+4.605229214"
Sep 17 00:30:09.581835 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:09.581597    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:09.582379 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:09.581940    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:09.661937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:09.661902    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal" event={"ID":"959cb07438d5c8900b2bb472409b47d9","Type":"ContainerStarted","Data":"78feb746cd8494a5dae2a4fa4aa9f51ef285c2e4d6f9ab4db7e0021819097b6f"}
Sep 17 00:30:10.099253 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:10.099218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:10.099427 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.099365    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:10.099606 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.099430    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs podName:06683d51-91e1-4e99-a67d-75d8a715b5ab nodeName:}" failed. No retries permitted until 2026-09-17 00:30:14.099412055 +0000 UTC m=+10.055008389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs") pod "network-metrics-daemon-w9nj2" (UID: "06683d51-91e1-4e99-a67d-75d8a715b5ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:10.200470 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:10.200440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:10.200631 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.200587    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:30:10.200631 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.200608    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:30:10.200631 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.200620    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zfvd5 for pod openshift-network-diagnostics/network-check-target-sdhcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:10.200777 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.200676    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5 podName:009bd9cc-b8a9-4c68-9a8e-158b348866cb nodeName:}" failed. No retries permitted until 2026-09-17 00:30:14.200657797 +0000 UTC m=+10.156254122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfvd5" (UniqueName: "kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5") pod "network-check-target-sdhcm" (UID: "009bd9cc-b8a9-4c68-9a8e-158b348866cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:10.301607 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:10.301496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:10.301759 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.301650    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:10.301759 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.301718    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret podName:289b9bc3-1131-473a-a889-0cdb22286663 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:14.301699132 +0000 UTC m=+10.257295454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret") pod "global-pull-secret-syncer-mlq9c" (UID: "289b9bc3-1131-473a-a889-0cdb22286663") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:10.583316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:10.583236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:10.583705 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.583364    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:10.583759 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:10.583740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:10.583865 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:10.583843    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:11.582406 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:11.582378    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:11.582588 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:11.582483    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:12.583246 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:12.582725    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:12.583246 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:12.582852    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:12.583787 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:12.583649    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:12.583787 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:12.583747    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:13.581643 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:13.581602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:13.581827 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:13.581722    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:14.133491 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:14.133456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:14.133978 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.133600    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:14.133978 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.133677    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs podName:06683d51-91e1-4e99-a67d-75d8a715b5ab nodeName:}" failed. No retries permitted until 2026-09-17 00:30:22.13365648 +0000 UTC m=+18.089252802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs") pod "network-metrics-daemon-w9nj2" (UID: "06683d51-91e1-4e99-a67d-75d8a715b5ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:14.235069 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:14.234461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:14.235069 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.234690    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:30:14.235069 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.234710    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:30:14.235069 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.234723    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zfvd5 for pod openshift-network-diagnostics/network-check-target-sdhcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:14.235069 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.234775    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5 podName:009bd9cc-b8a9-4c68-9a8e-158b348866cb nodeName:}" failed. No retries permitted until 2026-09-17 00:30:22.234761565 +0000 UTC m=+18.190357898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfvd5" (UniqueName: "kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5") pod "network-check-target-sdhcm" (UID: "009bd9cc-b8a9-4c68-9a8e-158b348866cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:14.335648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:14.335609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:14.335833 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.335759    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:14.335833 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.335824    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret podName:289b9bc3-1131-473a-a889-0cdb22286663 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:22.335806448 +0000 UTC m=+18.291402771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret") pod "global-pull-secret-syncer-mlq9c" (UID: "289b9bc3-1131-473a-a889-0cdb22286663") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:14.582070 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:14.581977    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:14.582255 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.582098    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:14.582255 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:14.582178    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:14.582368 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:14.582298    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:15.581354 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:15.581321    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:15.581780 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:15.581458    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:16.585312 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:16.585208    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:16.585771 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:16.585329    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:16.585771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:16.585714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:16.585896 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:16.585872    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:17.582209 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:17.582171    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:17.582392 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:17.582287    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:18.584397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:18.584367    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:18.584830 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:18.584410    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:18.584830 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:18.584508    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:18.584830 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:18.584609    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:19.581756 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:19.581710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:19.581960 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:19.581851    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:20.582349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:20.582311    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:20.582818 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:20.582326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:20.582818 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:20.582459    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:20.582818 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:20.582571    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:21.582138 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:21.582100    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:21.582326 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:21.582226    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:22.203232 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:22.203194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:22.203603 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.203373    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:22.203603 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.203449    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs podName:06683d51-91e1-4e99-a67d-75d8a715b5ab nodeName:}" failed. No retries permitted until 2026-09-17 00:30:38.203428089 +0000 UTC m=+34.159024411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs") pod "network-metrics-daemon-w9nj2" (UID: "06683d51-91e1-4e99-a67d-75d8a715b5ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:22.304373 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:22.304329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:22.304560 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.304537    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:30:22.304626 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.304566    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:30:22.304626 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.304582    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zfvd5 for pod openshift-network-diagnostics/network-check-target-sdhcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:22.304725 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.304646    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5 podName:009bd9cc-b8a9-4c68-9a8e-158b348866cb nodeName:}" failed. No retries permitted until 2026-09-17 00:30:38.304628887 +0000 UTC m=+34.260225207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfvd5" (UniqueName: "kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5") pod "network-check-target-sdhcm" (UID: "009bd9cc-b8a9-4c68-9a8e-158b348866cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:22.405627 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:22.405580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:22.405826 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.405772    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:22.405880 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.405850    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret podName:289b9bc3-1131-473a-a889-0cdb22286663 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:38.405829343 +0000 UTC m=+34.361425668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret") pod "global-pull-secret-syncer-mlq9c" (UID: "289b9bc3-1131-473a-a889-0cdb22286663") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:22.581425 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:22.581389    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:22.581599 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.581558    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:22.581654 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:22.581602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:22.581750 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:22.581726    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:23.582082 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:23.582044    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:23.582556 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:23.582193    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:24.582592 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:24.582508    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:24.583061 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:24.582637    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:24.583061 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:24.582677    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:24.583061 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:24.582770    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:25.582222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.582044    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:25.582343 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:25.582300    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:25.694621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.694578    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wr6h" event={"ID":"59884e4c-8c0b-4b73-9c4a-b9d0490ad534","Type":"ContainerStarted","Data":"30f1b9a409138f0e2e81424d80f7405089b4477ccdbd2ecc3b6cd20ac72d46f6"}
Sep 17 00:30:25.696230 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.696203    2576 generic.go:356] "Generic (PLEG): container finished" podID="50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8" containerID="e5ff9b45420c4deb24b7fcd7b7340127ea7f192a15974e580daa0aa33c3fbbda" exitCode=0
Sep 17 00:30:25.696347 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.696276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerDied","Data":"e5ff9b45420c4deb24b7fcd7b7340127ea7f192a15974e580daa0aa33c3fbbda"}
Sep 17 00:30:25.698026 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.697998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66" event={"ID":"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5","Type":"ContainerStarted","Data":"fa461698aa982759343e08d551ae860ebf98784a45a88b1800ef081f841c76f2"}
Sep 17 00:30:25.700273 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.700251    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:30:25.700650 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.700630    2576 generic.go:356] "Generic (PLEG): container finished" podID="116ae241-a23d-4780-ae66-a3ba9aee4499" containerID="90c8ac4a0cb810c47fed14101886238e393225cbf62fd71f712619681edf6b7d" exitCode=1
Sep 17 00:30:25.700729 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.700697    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerStarted","Data":"76d22bced02a2b1d3cd77cdec2f938022c6489102a99c6b54df65da74b0b6d25"}
Sep 17 00:30:25.700805 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.700787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerDied","Data":"90c8ac4a0cb810c47fed14101886238e393225cbf62fd71f712619681edf6b7d"}
Sep 17 00:30:25.700858 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.700850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerStarted","Data":"9069bcde4ffca56fef165716e7e9a7e6379c788ba6c482c1183cbcee5fdcfad3"}
Sep 17 00:30:25.702298 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.702275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rdtmd" event={"ID":"28bc9500-7f38-4e77-98c7-86001cac7fd9","Type":"ContainerStarted","Data":"5cb140c6afe804f798e5701a16966c61966c9babb661b0fabb3fb9bf301b849b"}
Sep 17 00:30:25.703997 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.703975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cf4c4" event={"ID":"42d25648-164b-4ad8-9d6e-d38f1f3301e4","Type":"ContainerStarted","Data":"5b25f6b89278b9d279e7c26a4dd7085dd8d61e5128cd36b0ad8c759e8a17864d"}
Sep 17 00:30:25.705663 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.705613    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vhmzk" event={"ID":"c99ec850-d4b6-4541-90a4-d158112c2570","Type":"ContainerStarted","Data":"f95bca84faebefc04292881bdd5f373690d4d0ad7e3c4790ddd4a761c9046e08"}
Sep 17 00:30:25.708424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.707364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6v7j7" event={"ID":"1787d2e6-4f22-41d1-bee6-d87f15c5936d","Type":"ContainerStarted","Data":"beddfa97ad10422a85a857a9299a40c53214332a08e771f814e7966c2c7e8884"}
Sep 17 00:30:25.708424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.708132    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4wr6h" podStartSLOduration=8.218527693 podStartE2EDuration="21.708118341s" totalImagesPullingTime="13.489590648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:25.707513712 +0000 UTC m=+21.663110054" watchObservedRunningTime="2026-09-17 00:30:25.708118341 +0000 UTC m=+21.663714682"
Sep 17 00:30:25.708424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.708320    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-51.ec2.internal" podStartSLOduration=19.708313775 podStartE2EDuration="19.708313775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:09.674818226 +0000 UTC m=+5.630414568" watchObservedRunningTime="2026-09-17 00:30:25.708313775 +0000 UTC m=+21.663910116"
Sep 17 00:30:25.721696 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.721653    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6v7j7" podStartSLOduration=4.163614238 podStartE2EDuration="21.72163962s" totalImagesPullingTime="17.558025382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:25.72138764 +0000 UTC m=+21.676983981" watchObservedRunningTime="2026-09-17 00:30:25.72163962 +0000 UTC m=+21.677235960"
Sep 17 00:30:25.733987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.733934    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cf4c4" podStartSLOduration=4.531080225 podStartE2EDuration="21.73391271s" totalImagesPullingTime="17.202832485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:25.733587999 +0000 UTC m=+21.689184340" watchObservedRunningTime="2026-09-17 00:30:25.73391271 +0000 UTC m=+21.689509052"
Sep 17 00:30:25.779288 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.779247    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vhmzk" podStartSLOduration=4.27153923 podStartE2EDuration="21.779234389s" totalImagesPullingTime="17.507695159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:25.763409659 +0000 UTC m=+21.719006000" watchObservedRunningTime="2026-09-17 00:30:25.779234389 +0000 UTC m=+21.734830729"
Sep 17 00:30:25.779381 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.779338    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rdtmd" podStartSLOduration=4.178736091 podStartE2EDuration="21.779335297s" totalImagesPullingTime="17.600599206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:25.77893064 +0000 UTC m=+21.734526980" watchObservedRunningTime="2026-09-17 00:30:25.779335297 +0000 UTC m=+21.734931638"
Sep 17 00:30:25.914647 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.914473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:25.996847 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:25.996678    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:26.581977 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.581954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:26.582119 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:26.582062    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:26.582119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.582095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:26.582257 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:26.582235    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:26.711164 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.711115    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rv25s" event={"ID":"3774f212-bf63-418d-86af-17d31c4adcc9","Type":"ContainerStarted","Data":"8577b5819c6f5cd4062b28e8ee1b909c1e1438473c73278e1dae7b1c7a03d328"}
Sep 17 00:30:26.713841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.713821    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:30:26.714246 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.714220    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerStarted","Data":"eb22b22effaf99031844586208ca7e8403cb6e7128ea1256fb447eb605b24142"}
Sep 17 00:30:26.714329 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.714291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerStarted","Data":"12d440177ccf83b30e7b91927fbb02f5eee64da4a08b17bb25dae8e9db4f1cc2"}
Sep 17 00:30:26.714329 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.714321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerStarted","Data":"818daaaac20ae5d948b4c15c415a756c69cb9ac58050c2246a260ebc38543ec7"}
Sep 17 00:30:26.724869 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.724835    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rv25s" podStartSLOduration=5.228840798 podStartE2EDuration="22.724823874s" totalImagesPullingTime="17.495983076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:26.724738533 +0000 UTC m=+22.680334882" watchObservedRunningTime="2026-09-17 00:30:26.724823874 +0000 UTC m=+22.680420237"
Sep 17 00:30:26.846540 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.846481    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:26.899001 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.898968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:26.915909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.915877    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 00:30:26.991456 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:26.991413    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:27.529199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:27.529075    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T00:30:26.915898052Z","UUID":"2412cb7b-e4ec-43e2-969c-fab3450a25ad","Handler":null,"Name":"","Endpoint":""}
Sep 17 00:30:27.531986 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:27.531961    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 00:30:27.532109 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:27.532017    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 00:30:27.581199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:27.581165    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:27.581377 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:27.581272    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:27.719095 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:27.719047    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66" event={"ID":"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5","Type":"ContainerStarted","Data":"f1d342df863a65698adbe6e2c4ea5800c41d0e84b14dcada1f75d1203ac0fe9b"}
Sep 17 00:30:28.581510 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:28.581476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:28.581674 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:28.581601    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:28.581674 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:28.581646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:28.581776 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:28.581737    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:28.724940 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:28.724884    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66" event={"ID":"4a4a2e00-6841-4a9b-ab3d-f86001cf95f5","Type":"ContainerStarted","Data":"6fca87551b25823c6ac497235808be2d3b16f9a068354dd4185d418fe47a24e4"}
Sep 17 00:30:28.728994 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:28.728974    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:30:28.729436 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:28.729413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerStarted","Data":"80c019379c913a69c0c49803e453f4f4c9fed8e654ffda69e1fcb8ea893b4978"}
Sep 17 00:30:28.740836 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:28.740777    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5cd66" podStartSLOduration=4.720038985 podStartE2EDuration="24.740759584s" totalImagesPullingTime="20.020720599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:28.739887051 +0000 UTC m=+24.695483393" watchObservedRunningTime="2026-09-17 00:30:28.740759584 +0000 UTC m=+24.696355927"
Sep 17 00:30:28.748373 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:28.748352    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:28.748995 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:28.748976    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:29.582030 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:29.581996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:29.582237 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:29.582123    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:29.731810 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:29.731713    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:29.732255 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:29.732026    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4wr6h"
Sep 17 00:30:29.786362 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:29.786329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:30.475959 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:30.475920    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:30.530118 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:30.530000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:30.585893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:30.585857    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:30.585893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:30.585898    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:30.586108 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:30.585965    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:30.586108 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:30.586085    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:31.153738 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.153510    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:31.210458 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.210426    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:31.456919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.456838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:31.581374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.581342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:31.581546 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:31.581450    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:31.739829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.739759    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:30:31.740183 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.740157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerStarted","Data":"972614fdd3c8ea38d54ee5cb124a6f3c27d6940b3b671360886a3e39dc9e7eb0"}
Sep 17 00:30:31.740359 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.740340    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:31.740493 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.740407    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:31.740493 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.740456    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:31.740747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.740722    2576 scope.go:117] "RemoveContainer" containerID="90c8ac4a0cb810c47fed14101886238e393225cbf62fd71f712619681edf6b7d"
Sep 17 00:30:31.742930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.742475    2576 generic.go:356] "Generic (PLEG): container finished" podID="50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8" containerID="d676b18bc891983faf34d4bf99935dd518d7350f490a35a87c4e7225d19d36c0" exitCode=0
Sep 17 00:30:31.742930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.742502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerDied","Data":"d676b18bc891983faf34d4bf99935dd518d7350f490a35a87c4e7225d19d36c0"}
Sep 17 00:30:31.757705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.757687    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:31.757928 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.757913    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:30:31.814601 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.814569    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:31.870838 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:31.870814    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:32.586929 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.586124    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:32.586929 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:32.586496    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:32.586929 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.586315    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:32.586929 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:32.586613    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:32.617924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.617891    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdhcm"]
Sep 17 00:30:32.618048 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.618018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:32.618118 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:32.618100    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:32.620836 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.620815    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlq9c"]
Sep 17 00:30:32.621296 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.621277    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w9nj2"]
Sep 17 00:30:32.747400 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.747369    2576 generic.go:356] "Generic (PLEG): container finished" podID="50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8" containerID="3d802500059db54a43c94b2a20a712ec274124578dea1612e686923e5574e38b" exitCode=0
Sep 17 00:30:32.747569 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.747469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerDied","Data":"3d802500059db54a43c94b2a20a712ec274124578dea1612e686923e5574e38b"}
Sep 17 00:30:32.751806 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.751790    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:30:32.752198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.752176    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" event={"ID":"116ae241-a23d-4780-ae66-a3ba9aee4499","Type":"ContainerStarted","Data":"459eaf843bfff848cdd4458cec0d08989e1193104c2151cdfceb9cc001aad58f"}
Sep 17 00:30:32.752311 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.752200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:32.752311 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:32.752213    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:32.752311 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:32.752290    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:32.752500 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:32.752481    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:33.756974 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:33.756947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerStarted","Data":"749863c3866e840e9b228f15f90edc477c4d28cfb131001ae2e73af024403ed8"}
Sep 17 00:30:33.776766 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:33.776724    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff" podStartSLOduration=12.199568192 podStartE2EDuration="29.776708458s" totalImagesPullingTime="17.577140266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:32.790351954 +0000 UTC m=+28.745948295" watchObservedRunningTime="2026-09-17 00:30:33.776708458 +0000 UTC m=+29.732304781"
Sep 17 00:30:34.393385 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:34.393352    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:34.582559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:34.582534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:34.582705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:34.582630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:34.582705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:34.582672    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:34.582705 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:34.582693    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:34.582853 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:34.582714    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:34.582853 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:34.582772    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:34.598230 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:34.598203    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:34.695840 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:34.695761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:34.761653 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:34.761618    2576 generic.go:356] "Generic (PLEG): container finished" podID="50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8" containerID="749863c3866e840e9b228f15f90edc477c4d28cfb131001ae2e73af024403ed8" exitCode=0
Sep 17 00:30:34.762029 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:34.761688    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerDied","Data":"749863c3866e840e9b228f15f90edc477c4d28cfb131001ae2e73af024403ed8"}
Sep 17 00:30:36.581627 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:36.581596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:36.582180 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:36.581596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:36.582180 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:36.581716    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdhcm" podUID="009bd9cc-b8a9-4c68-9a8e-158b348866cb"
Sep 17 00:30:36.582180 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:36.581739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:36.582180 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:36.581849    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlq9c" podUID="289b9bc3-1131-473a-a889-0cdb22286663"
Sep 17 00:30:36.582180 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:36.581963    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:30:38.226257 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.226171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:38.226742 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.226304    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:38.226742 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.226364    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs podName:06683d51-91e1-4e99-a67d-75d8a715b5ab nodeName:}" failed. No retries permitted until 2026-09-17 00:31:10.226349128 +0000 UTC m=+66.181945459 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs") pod "network-metrics-daemon-w9nj2" (UID: "06683d51-91e1-4e99-a67d-75d8a715b5ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:30:38.290794 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.290760    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-51.ec2.internal" event="NodeReady"
Sep 17 00:30:38.290956 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.290937    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 00:30:38.322490 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.322457    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d46c8d77b-f7fpj"]
Sep 17 00:30:38.326841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.326812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:38.327005 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.326989    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:30:38.327065 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.327012    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:30:38.327065 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.327024    2576 projected.go:196] Error preparing data for projected volume kube-api-access-zfvd5 for pod openshift-network-diagnostics/network-check-target-sdhcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:38.327180 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.327068    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5 podName:009bd9cc-b8a9-4c68-9a8e-158b348866cb nodeName:}" failed. No retries permitted until 2026-09-17 00:31:10.327055291 +0000 UTC m=+66.282651629 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfvd5" (UniqueName: "kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5") pod "network-check-target-sdhcm" (UID: "009bd9cc-b8a9-4c68-9a8e-158b348866cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:30:38.327180 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.327094    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.329481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.329459    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 00:30:38.329624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.329602    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cjf6n\""
Sep 17 00:30:38.329736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.329681    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 00:30:38.329849 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.329834    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 00:30:38.331887 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.331870    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x6lxh"]
Sep 17 00:30:38.335622 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.335446    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r4k48"]
Sep 17 00:30:38.335727 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.335654    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x6lxh"
Sep 17 00:30:38.339955 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.338726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 00:30:38.339955 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.339041    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 00:30:38.339955 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.339047    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 00:30:38.339955 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.339461    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6wf2d\""
Sep 17 00:30:38.341790 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.341757    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 00:30:38.344362 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.344254    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d46c8d77b-f7fpj"]
Sep 17 00:30:38.344362 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.344344    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.345934 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.345912    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r4k48"]
Sep 17 00:30:38.346389 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.346370    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 00:30:38.346676 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.346655    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n9svt\""
Sep 17 00:30:38.346803 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.346664    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 00:30:38.350928 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.350906    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x6lxh"]
Sep 17 00:30:38.427351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427314    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.427351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9ht\" (UniqueName: \"kubernetes.io/projected/bc042542-1f09-45d0-b848-cbf7f11ff4e7-kube-api-access-sg9ht\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.427600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/853a6578-a7f7-4ed0-9331-e9877dadf9ce-ca-trust-extracted\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.427600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-trusted-ca\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.427600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-installation-pull-secrets\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.427600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc-cert\") pod \"ingress-canary-x6lxh\" (UID: \"f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc\") " pod="openshift-ingress-canary/ingress-canary-x6lxh"
Sep 17 00:30:38.427798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-image-registry-private-configuration\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.427798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-bound-sa-token\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.427798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc042542-1f09-45d0-b848-cbf7f11ff4e7-tmp-dir\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.427798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.427798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gndj\" (UniqueName: \"kubernetes.io/projected/f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc-kube-api-access-8gndj\") pod \"ingress-canary-x6lxh\" (UID: \"f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc\") " pod="openshift-ingress-canary/ingress-canary-x6lxh"
Sep 17 00:30:38.427798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc042542-1f09-45d0-b848-cbf7f11ff4e7-config-volume\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.427798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-certificates\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.427798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7kqk\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-kube-api-access-q7kqk\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.428132 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.427850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:38.428132 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.427968    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:38.428132 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.428019    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret podName:289b9bc3-1131-473a-a889-0cdb22286663 nodeName:}" failed. No retries permitted until 2026-09-17 00:31:10.428005561 +0000 UTC m=+66.383601880 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret") pod "global-pull-secret-syncer-mlq9c" (UID: "289b9bc3-1131-473a-a889-0cdb22286663") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:30:38.450612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.450570    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:38.500643 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.500562    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:38.528559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.528531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7kqk\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-kube-api-access-q7kqk\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.528715 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.528606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.528715 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.528650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9ht\" (UniqueName: \"kubernetes.io/projected/bc042542-1f09-45d0-b848-cbf7f11ff4e7-kube-api-access-sg9ht\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.528715 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.528670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/853a6578-a7f7-4ed0-9331-e9877dadf9ce-ca-trust-extracted\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.528715 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.528691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-trusted-ca\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.528715 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.528714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-installation-pull-secrets\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.528964 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.528798    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:30:38.528964 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.528886    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls podName:bc042542-1f09-45d0-b848-cbf7f11ff4e7 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:39.028863524 +0000 UTC m=+34.984459847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls") pod "dns-default-r4k48" (UID: "bc042542-1f09-45d0-b848-cbf7f11ff4e7") : secret "dns-default-metrics-tls" not found
Sep 17 00:30:38.528964 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.528909    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc-cert\") pod \"ingress-canary-x6lxh\" (UID: \"f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc\") " pod="openshift-ingress-canary/ingress-canary-x6lxh"
Sep 17 00:30:38.529126 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-image-registry-private-configuration\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.529126 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-bound-sa-token\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.529256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc042542-1f09-45d0-b848-cbf7f11ff4e7-tmp-dir\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.529256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.529256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gndj\" (UniqueName: \"kubernetes.io/projected/f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc-kube-api-access-8gndj\") pod \"ingress-canary-x6lxh\" (UID: \"f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc\") " pod="openshift-ingress-canary/ingress-canary-x6lxh"
Sep 17 00:30:38.529256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc042542-1f09-45d0-b848-cbf7f11ff4e7-config-volume\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.529428 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.529327    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:30:38.529428 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.529343    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d46c8d77b-f7fpj: secret "image-registry-tls" not found
Sep 17 00:30:38.529428 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:38.529401    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls podName:853a6578-a7f7-4ed0-9331-e9877dadf9ce nodeName:}" failed. No retries permitted until 2026-09-17 00:30:39.029382811 +0000 UTC m=+34.984979132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls") pod "image-registry-5d46c8d77b-f7fpj" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce") : secret "image-registry-tls" not found
Sep 17 00:30:38.529626 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-certificates\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.529678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529645    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc042542-1f09-45d0-b848-cbf7f11ff4e7-tmp-dir\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.529818 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/853a6578-a7f7-4ed0-9331-e9877dadf9ce-ca-trust-extracted\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.529884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529867    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc042542-1f09-45d0-b848-cbf7f11ff4e7-config-volume\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.529937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.529906    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-trusted-ca\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.530163 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.530127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-certificates\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.533433 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.533413    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-image-registry-private-configuration\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.533433 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.533421    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc-cert\") pod \"ingress-canary-x6lxh\" (UID: \"f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc\") " pod="openshift-ingress-canary/ingress-canary-x6lxh"
Sep 17 00:30:38.533593 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.533413    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-installation-pull-secrets\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.541330 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.541175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gndj\" (UniqueName: \"kubernetes.io/projected/f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc-kube-api-access-8gndj\") pod \"ingress-canary-x6lxh\" (UID: \"f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc\") " pod="openshift-ingress-canary/ingress-canary-x6lxh"
Sep 17 00:30:38.541575 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.541509    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7kqk\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-kube-api-access-q7kqk\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.541689 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.541574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-bound-sa-token\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:38.543096 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.543065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9ht\" (UniqueName: \"kubernetes.io/projected/bc042542-1f09-45d0-b848-cbf7f11ff4e7-kube-api-access-sg9ht\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:38.586549 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.586519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:30:38.586728 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.586638    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:30:38.586728 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.586673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:30:38.589195 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.589168    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 00:30:38.589333 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.589216    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lqlnk\""
Sep 17 00:30:38.589333 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.589169    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 00:30:38.589333 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.589317    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 00:30:38.589529 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.589467    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 00:30:38.589585 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.589548    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plv5t\""
Sep 17 00:30:38.652584 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:38.652558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x6lxh"
Sep 17 00:30:39.034430 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:39.034395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:39.034636 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:39.034499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:39.034636 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:39.034548    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:30:39.034636 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:39.034571    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d46c8d77b-f7fpj: secret "image-registry-tls" not found
Sep 17 00:30:39.034775 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:39.034644    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls podName:853a6578-a7f7-4ed0-9331-e9877dadf9ce nodeName:}" failed. No retries permitted until 2026-09-17 00:30:40.034622449 +0000 UTC m=+35.990218813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls") pod "image-registry-5d46c8d77b-f7fpj" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce") : secret "image-registry-tls" not found
Sep 17 00:30:39.034775 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:39.034646    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:30:39.034775 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:39.034724    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls podName:bc042542-1f09-45d0-b848-cbf7f11ff4e7 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:40.034704449 +0000 UTC m=+35.990300795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls") pod "dns-default-r4k48" (UID: "bc042542-1f09-45d0-b848-cbf7f11ff4e7") : secret "dns-default-metrics-tls" not found
Sep 17 00:30:39.780790 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:39.780762    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerStarted","Data":"f7050c143003a5867875556dedd6ef1a3db4b90e57ea213847c950d89eaca84b"}
Sep 17 00:30:39.790319 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:39.790300    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x6lxh"]
Sep 17 00:30:39.812100 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:30:39.812080    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c1d968_4bc7_4e1b_acea_1388a6a0ffdc.slice/crio-b4d79b972cec7d1d0373eca0487805c45a76b96ce00a34f5ad5f92c38bdee9ef WatchSource:0}: Error finding container b4d79b972cec7d1d0373eca0487805c45a76b96ce00a34f5ad5f92c38bdee9ef: Status 404 returned error can't find the container with id b4d79b972cec7d1d0373eca0487805c45a76b96ce00a34f5ad5f92c38bdee9ef
Sep 17 00:30:40.043546 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:40.043463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:40.043698 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:40.043558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:40.043698 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:40.043620    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:30:40.043698 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:40.043636    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d46c8d77b-f7fpj: secret "image-registry-tls" not found
Sep 17 00:30:40.043698 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:40.043640    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:30:40.043830 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:40.043705    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls podName:853a6578-a7f7-4ed0-9331-e9877dadf9ce nodeName:}" failed. No retries permitted until 2026-09-17 00:30:42.043685876 +0000 UTC m=+37.999282208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls") pod "image-registry-5d46c8d77b-f7fpj" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce") : secret "image-registry-tls" not found
Sep 17 00:30:40.043830 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:40.043720    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls podName:bc042542-1f09-45d0-b848-cbf7f11ff4e7 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:42.043713072 +0000 UTC m=+37.999309392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls") pod "dns-default-r4k48" (UID: "bc042542-1f09-45d0-b848-cbf7f11ff4e7") : secret "dns-default-metrics-tls" not found
Sep 17 00:30:40.784433 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:40.784395    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x6lxh" event={"ID":"f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc","Type":"ContainerStarted","Data":"b4d79b972cec7d1d0373eca0487805c45a76b96ce00a34f5ad5f92c38bdee9ef"}
Sep 17 00:30:40.787676 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:40.787643    2576 generic.go:356] "Generic (PLEG): container finished" podID="50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8" containerID="f7050c143003a5867875556dedd6ef1a3db4b90e57ea213847c950d89eaca84b" exitCode=0
Sep 17 00:30:40.787804 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:40.787679    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerDied","Data":"f7050c143003a5867875556dedd6ef1a3db4b90e57ea213847c950d89eaca84b"}
Sep 17 00:30:41.794293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:41.793998    2576 generic.go:356] "Generic (PLEG): container finished" podID="50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8" containerID="3ed4faf3245cd90321057ed941f0fb73a75f4ef27ff7f2d842e4ad1377bbcbbe" exitCode=0
Sep 17 00:30:41.794293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:41.794084    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerDied","Data":"3ed4faf3245cd90321057ed941f0fb73a75f4ef27ff7f2d842e4ad1377bbcbbe"}
Sep 17 00:30:42.060849 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:42.060764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:42.060849 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:42.060827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:42.061072 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:42.060901    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:30:42.061072 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:42.060945    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:30:42.061072 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:42.060959    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d46c8d77b-f7fpj: secret "image-registry-tls" not found
Sep 17 00:30:42.061072 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:42.060978    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls podName:bc042542-1f09-45d0-b848-cbf7f11ff4e7 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:46.060961093 +0000 UTC m=+42.016557436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls") pod "dns-default-r4k48" (UID: "bc042542-1f09-45d0-b848-cbf7f11ff4e7") : secret "dns-default-metrics-tls" not found
Sep 17 00:30:42.061072 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:42.061006    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls podName:853a6578-a7f7-4ed0-9331-e9877dadf9ce nodeName:}" failed. No retries permitted until 2026-09-17 00:30:46.060991166 +0000 UTC m=+42.016587504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls") pod "image-registry-5d46c8d77b-f7fpj" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce") : secret "image-registry-tls" not found
Sep 17 00:30:42.468256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:42.468227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:42.568634 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:42.568608    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:42.800174 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:42.800130    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84cm9" event={"ID":"50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8","Type":"ContainerStarted","Data":"cc544491adce11f98e8378be90a073c67bb7dfed87d2ee723731c8f213b88b23"}
Sep 17 00:30:42.801599 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:42.801579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x6lxh" event={"ID":"f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc","Type":"ContainerStarted","Data":"a943fb6fa0eb74859274c851519a7477384bd619d9e7efa4894132dc601dafff"}
Sep 17 00:30:42.820408 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:42.820364    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-84cm9" podStartSLOduration=9.75478967 podStartE2EDuration="38.820351469s" totalImagesPullingTime="29.065561799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:42.818671923 +0000 UTC m=+38.774268288" watchObservedRunningTime="2026-09-17 00:30:42.820351469 +0000 UTC m=+38.775947809"
Sep 17 00:30:42.832014 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:42.831968    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x6lxh" podStartSLOduration=2.182406502 podStartE2EDuration="4.831955992s" totalImagesPullingTime="2.64954949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:42.831503873 +0000 UTC m=+38.787100214" watchObservedRunningTime="2026-09-17 00:30:42.831955992 +0000 UTC m=+38.787552332"
Sep 17 00:30:42.943349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:42.943314    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:43.044500 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:43.044467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:43.748021 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:43.747985    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:43.797759 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:43.797728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:46.090275 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:46.090239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:46.090739 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:46.090282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:46.090739 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:46.090369    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:30:46.090739 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:46.090381    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d46c8d77b-f7fpj: secret "image-registry-tls" not found
Sep 17 00:30:46.090739 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:46.090418    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:30:46.090739 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:46.090432    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls podName:853a6578-a7f7-4ed0-9331-e9877dadf9ce nodeName:}" failed. No retries permitted until 2026-09-17 00:30:54.090418416 +0000 UTC m=+50.046014747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls") pod "image-registry-5d46c8d77b-f7fpj" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce") : secret "image-registry-tls" not found
Sep 17 00:30:46.090739 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:46.090499    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls podName:bc042542-1f09-45d0-b848-cbf7f11ff4e7 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:54.090480842 +0000 UTC m=+50.046077161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls") pod "dns-default-r4k48" (UID: "bc042542-1f09-45d0-b848-cbf7f11ff4e7") : secret "dns-default-metrics-tls" not found
Sep 17 00:30:46.836105 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:46.836057    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:46.888192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:46.888132    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:48.802988 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:48.802954    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:48.849243 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:48.849216    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:49.540341 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:49.540309    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:49.592804 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:49.592765    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:30:54.150553 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:54.150516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:30:54.150553 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:30:54.150561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:30:54.150980 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:54.150666    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:30:54.150980 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:54.150670    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:30:54.150980 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:54.150742    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls podName:bc042542-1f09-45d0-b848-cbf7f11ff4e7 nodeName:}" failed. No retries permitted until 2026-09-17 00:31:10.150724816 +0000 UTC m=+66.106321154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls") pod "dns-default-r4k48" (UID: "bc042542-1f09-45d0-b848-cbf7f11ff4e7") : secret "dns-default-metrics-tls" not found
Sep 17 00:30:54.150980 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:54.150677    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d46c8d77b-f7fpj: secret "image-registry-tls" not found
Sep 17 00:30:54.150980 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:30:54.150807    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls podName:853a6578-a7f7-4ed0-9331-e9877dadf9ce nodeName:}" failed. No retries permitted until 2026-09-17 00:31:10.150795582 +0000 UTC m=+66.106391919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls") pod "image-registry-5d46c8d77b-f7fpj" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce") : secret "image-registry-tls" not found
Sep 17 00:31:03.768183 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:03.768139    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tfff"
Sep 17 00:31:03.921076 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:03.921048    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:31:04.013871 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:04.013838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:31:07.472008 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:07.471962    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:31:07.573072 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:07.573044    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:31:07.625896 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:07.625849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:31:10.162240 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.162202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:31:10.162736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.162250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:31:10.162736 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:10.162359    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:31:10.162736 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:10.162363    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:31:10.162736 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:10.162381    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d46c8d77b-f7fpj: secret "image-registry-tls" not found
Sep 17 00:31:10.162736 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:10.162444    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls podName:bc042542-1f09-45d0-b848-cbf7f11ff4e7 nodeName:}" failed. No retries permitted until 2026-09-17 00:31:42.16242543 +0000 UTC m=+98.118021763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls") pod "dns-default-r4k48" (UID: "bc042542-1f09-45d0-b848-cbf7f11ff4e7") : secret "dns-default-metrics-tls" not found
Sep 17 00:31:10.162736 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:10.162461    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls podName:853a6578-a7f7-4ed0-9331-e9877dadf9ce nodeName:}" failed. No retries permitted until 2026-09-17 00:31:42.162452576 +0000 UTC m=+98.118048896 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls") pod "image-registry-5d46c8d77b-f7fpj" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce") : secret "image-registry-tls" not found
Sep 17 00:31:10.262849 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.262810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:31:10.265429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.265409    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 00:31:10.273554 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:10.273535    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 00:31:10.273644 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:10.273602    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs podName:06683d51-91e1-4e99-a67d-75d8a715b5ab nodeName:}" failed. No retries permitted until 2026-09-17 00:32:14.273580644 +0000 UTC m=+130.229176982 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs") pod "network-metrics-daemon-w9nj2" (UID: "06683d51-91e1-4e99-a67d-75d8a715b5ab") : secret "metrics-daemon-secret" not found
Sep 17 00:31:10.363799 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.363763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:31:10.366477 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.366457    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 00:31:10.376259 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.376243    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 00:31:10.389077 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.389048    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvd5\" (UniqueName: \"kubernetes.io/projected/009bd9cc-b8a9-4c68-9a8e-158b348866cb-kube-api-access-zfvd5\") pod \"network-check-target-sdhcm\" (UID: \"009bd9cc-b8a9-4c68-9a8e-158b348866cb\") " pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:31:10.412295 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.412220    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lqlnk\""
Sep 17 00:31:10.420215 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.420192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:31:10.464682 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.464652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:31:10.467189 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.467166    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 00:31:10.477394 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.477372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/289b9bc3-1131-473a-a889-0cdb22286663-original-pull-secret\") pod \"global-pull-secret-syncer-mlq9c\" (UID: \"289b9bc3-1131-473a-a889-0cdb22286663\") " pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:31:10.698276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.698189    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlq9c"
Sep 17 00:31:10.726266 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.726236    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdhcm"]
Sep 17 00:31:10.729528 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:31:10.729503    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009bd9cc_b8a9_4c68_9a8e_158b348866cb.slice/crio-1c49dcb44e111d5d0815d714ee90453f47642b88bdc1f77e80342d9673b6dfa2 WatchSource:0}: Error finding container 1c49dcb44e111d5d0815d714ee90453f47642b88bdc1f77e80342d9673b6dfa2: Status 404 returned error can't find the container with id 1c49dcb44e111d5d0815d714ee90453f47642b88bdc1f77e80342d9673b6dfa2
Sep 17 00:31:10.865623 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.865588    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdhcm" event={"ID":"009bd9cc-b8a9-4c68-9a8e-158b348866cb","Type":"ContainerStarted","Data":"1c49dcb44e111d5d0815d714ee90453f47642b88bdc1f77e80342d9673b6dfa2"}
Sep 17 00:31:10.993581 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:10.993510    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlq9c"]
Sep 17 00:31:10.996932 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:31:10.996901    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289b9bc3_1131_473a_a889_0cdb22286663.slice/crio-d72e0dd7f8137dd84ae65b4576ae386efd2dba1149bced410f2dd62dada3aab8 WatchSource:0}: Error finding container d72e0dd7f8137dd84ae65b4576ae386efd2dba1149bced410f2dd62dada3aab8: Status 404 returned error can't find the container with id d72e0dd7f8137dd84ae65b4576ae386efd2dba1149bced410f2dd62dada3aab8
Sep 17 00:31:11.869648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:11.869609    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlq9c" event={"ID":"289b9bc3-1131-473a-a889-0cdb22286663","Type":"ContainerStarted","Data":"d72e0dd7f8137dd84ae65b4576ae386efd2dba1149bced410f2dd62dada3aab8"}
Sep 17 00:31:16.883841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:16.883796    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdhcm" event={"ID":"009bd9cc-b8a9-4c68-9a8e-158b348866cb","Type":"ContainerStarted","Data":"56a2e07f39659ac988808694cafce344697d22e905a659952c28a41f2c306d62"}
Sep 17 00:31:16.884347 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:16.883928    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:31:16.885432 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:16.885415    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlq9c" event={"ID":"289b9bc3-1131-473a-a889-0cdb22286663","Type":"ContainerStarted","Data":"e426b9818e09e84ed1790c3d5b38db54da39000bc6b256a5566b043f865479f1"}
Sep 17 00:31:16.898022 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:16.897974    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sdhcm" podStartSLOduration=67.759825168 podStartE2EDuration="1m12.897959058s" totalImagesPullingTime="5.13813389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:31:16.896983783 +0000 UTC m=+72.852580124" watchObservedRunningTime="2026-09-17 00:31:16.897959058 +0000 UTC m=+72.853555398"
Sep 17 00:31:16.914064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:16.914013    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mlq9c" podStartSLOduration=68.039115087 podStartE2EDuration="1m12.913998434s" totalImagesPullingTime="4.874883347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:31:16.913421944 +0000 UTC m=+72.869018285" watchObservedRunningTime="2026-09-17 00:31:16.913998434 +0000 UTC m=+72.869594817"
Sep 17 00:31:34.544814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.544675    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"]
Sep 17 00:31:34.548987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.548969    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"
Sep 17 00:31:34.552648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.552607    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-w9hw6\""
Sep 17 00:31:34.552648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.552607    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 00:31:34.552648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.552649    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 00:31:34.552894 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.552673    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 00:31:34.552894 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.552611    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 00:31:34.557222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.557199    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"]
Sep 17 00:31:34.637214 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.637193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgrrq\" (UniqueName: \"kubernetes.io/projected/2c2262ca-21ab-415b-9277-2f8de806811d-kube-api-access-zgrrq\") pod \"managed-serviceaccount-addon-agent-7c567547d6-gj7x8\" (UID: \"2c2262ca-21ab-415b-9277-2f8de806811d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"
Sep 17 00:31:34.637320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.637247    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c2262ca-21ab-415b-9277-2f8de806811d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c567547d6-gj7x8\" (UID: \"2c2262ca-21ab-415b-9277-2f8de806811d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"
Sep 17 00:31:34.737608 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.737577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgrrq\" (UniqueName: \"kubernetes.io/projected/2c2262ca-21ab-415b-9277-2f8de806811d-kube-api-access-zgrrq\") pod \"managed-serviceaccount-addon-agent-7c567547d6-gj7x8\" (UID: \"2c2262ca-21ab-415b-9277-2f8de806811d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"
Sep 17 00:31:34.737708 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.737630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c2262ca-21ab-415b-9277-2f8de806811d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c567547d6-gj7x8\" (UID: \"2c2262ca-21ab-415b-9277-2f8de806811d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"
Sep 17 00:31:34.740311 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.740291    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c2262ca-21ab-415b-9277-2f8de806811d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c567547d6-gj7x8\" (UID: \"2c2262ca-21ab-415b-9277-2f8de806811d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"
Sep 17 00:31:34.744770 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.744750    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgrrq\" (UniqueName: \"kubernetes.io/projected/2c2262ca-21ab-415b-9277-2f8de806811d-kube-api-access-zgrrq\") pod \"managed-serviceaccount-addon-agent-7c567547d6-gj7x8\" (UID: \"2c2262ca-21ab-415b-9277-2f8de806811d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"
Sep 17 00:31:34.871454 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:34.871402    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"
Sep 17 00:31:35.138530 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:35.138462    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8"]
Sep 17 00:31:35.141360 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:31:35.141336    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c2262ca_21ab_415b_9277_2f8de806811d.slice/crio-55d4a9142e8424bb35800f738964d29ae27186944a004c836c0e62cee6094a0a WatchSource:0}: Error finding container 55d4a9142e8424bb35800f738964d29ae27186944a004c836c0e62cee6094a0a: Status 404 returned error can't find the container with id 55d4a9142e8424bb35800f738964d29ae27186944a004c836c0e62cee6094a0a
Sep 17 00:31:35.934878 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:35.934836    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8" event={"ID":"2c2262ca-21ab-415b-9277-2f8de806811d","Type":"ContainerStarted","Data":"55d4a9142e8424bb35800f738964d29ae27186944a004c836c0e62cee6094a0a"}
Sep 17 00:31:40.949781 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:40.949745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8" event={"ID":"2c2262ca-21ab-415b-9277-2f8de806811d","Type":"ContainerStarted","Data":"c621e5facb341f81742f61a8ba9a4266edca7123e3879e4065a36e434d83606d"}
Sep 17 00:31:40.964744 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:40.964696    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8" podStartSLOduration=2.026243697 podStartE2EDuration="6.964682383s" totalImagesPullingTime="4.938438686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:31:40.964359364 +0000 UTC m=+96.919955706" watchObservedRunningTime="2026-09-17 00:31:40.964682383 +0000 UTC m=+96.920278723"
Sep 17 00:31:42.185113 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:42.185088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:31:42.185486 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:42.185132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:31:42.185486 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:42.185235    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:31:42.185486 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:42.185246    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d46c8d77b-f7fpj: secret "image-registry-tls" not found
Sep 17 00:31:42.185486 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:42.185257    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:31:42.185486 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:42.185300    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls podName:853a6578-a7f7-4ed0-9331-e9877dadf9ce nodeName:}" failed. No retries permitted until 2026-09-17 00:32:46.185285505 +0000 UTC m=+162.140881824 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls") pod "image-registry-5d46c8d77b-f7fpj" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce") : secret "image-registry-tls" not found
Sep 17 00:31:42.185486 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:31:42.185331    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls podName:bc042542-1f09-45d0-b848-cbf7f11ff4e7 nodeName:}" failed. No retries permitted until 2026-09-17 00:32:46.185308929 +0000 UTC m=+162.140905271 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls") pod "dns-default-r4k48" (UID: "bc042542-1f09-45d0-b848-cbf7f11ff4e7") : secret "dns-default-metrics-tls" not found
Sep 17 00:31:47.890731 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:31:47.890693    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sdhcm"
Sep 17 00:32:14.302659 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:32:14.302619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:32:14.303129 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:14.302736    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 00:32:14.303129 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:14.302802    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs podName:06683d51-91e1-4e99-a67d-75d8a715b5ab nodeName:}" failed. No retries permitted until 2026-09-17 00:34:16.302787097 +0000 UTC m=+252.258383430 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs") pod "network-metrics-daemon-w9nj2" (UID: "06683d51-91e1-4e99-a67d-75d8a715b5ab") : secret "metrics-daemon-secret" not found
Sep 17 00:32:41.344590 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:41.344539    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj" podUID="853a6578-a7f7-4ed0-9331-e9877dadf9ce"
Sep 17 00:32:41.358708 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:41.358672    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-r4k48" podUID="bc042542-1f09-45d0-b848-cbf7f11ff4e7"
Sep 17 00:32:41.604583 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:41.604509    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-w9nj2" podUID="06683d51-91e1-4e99-a67d-75d8a715b5ab"
Sep 17 00:32:42.092593 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:32:42.092562    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r4k48"
Sep 17 00:32:42.092747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:32:42.092562    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:32:46.217563 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:32:46.217522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:32:46.217563 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:32:46.217568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:32:46.218100 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:46.217671    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:32:46.218100 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:46.217684    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d46c8d77b-f7fpj: secret "image-registry-tls" not found
Sep 17 00:32:46.218100 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:46.217699    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:32:46.218100 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:46.217749    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls podName:853a6578-a7f7-4ed0-9331-e9877dadf9ce nodeName:}" failed. No retries permitted until 2026-09-17 00:34:48.217733334 +0000 UTC m=+284.173329731 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls") pod "image-registry-5d46c8d77b-f7fpj" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce") : secret "image-registry-tls" not found
Sep 17 00:32:46.218100 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:32:46.217797    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls podName:bc042542-1f09-45d0-b848-cbf7f11ff4e7 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:48.217774346 +0000 UTC m=+284.173370711 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls") pod "dns-default-r4k48" (UID: "bc042542-1f09-45d0-b848-cbf7f11ff4e7") : secret "dns-default-metrics-tls" not found
Sep 17 00:32:56.581677 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:32:56.581599    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:33:27.233354 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:27.233321    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:40.227188 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:40.227161    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c2262ca-21ab-415b-9277-2f8de806811d" containerID="c621e5facb341f81742f61a8ba9a4266edca7123e3879e4065a36e434d83606d" exitCode=255
Sep 17 00:33:40.227448 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:40.227217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8" event={"ID":"2c2262ca-21ab-415b-9277-2f8de806811d","Type":"ContainerDied","Data":"c621e5facb341f81742f61a8ba9a4266edca7123e3879e4065a36e434d83606d"}
Sep 17 00:33:40.227506 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:40.227490    2576 scope.go:117] "RemoveContainer" containerID="c621e5facb341f81742f61a8ba9a4266edca7123e3879e4065a36e434d83606d"
Sep 17 00:33:41.231465 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:41.231427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c567547d6-gj7x8" event={"ID":"2c2262ca-21ab-415b-9277-2f8de806811d","Type":"ContainerStarted","Data":"c139ef0261807e301c5b219f3a79cb29b648586b8d54387d3e4311e792604502"}
Sep 17 00:33:53.930798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.930759    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4"]
Sep 17 00:33:53.933781 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.933760    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4"
Sep 17 00:33:53.937315 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.937291    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"]
Sep 17 00:33:53.937315 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.937310    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 00:33:53.938130 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.937925    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jfmf6\""
Sep 17 00:33:53.938250 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.938077    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 00:33:53.942859 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.942835    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"]
Sep 17 00:33:53.943256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.943030    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:53.945809 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.945789    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 00:33:53.945913 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.945861    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 00:33:53.946016 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.945793    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 00:33:53.946016 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.946015    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-59g6x"]
Sep 17 00:33:53.946212 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.946068    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 00:33:53.946296 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.946278    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ggtt4\""
Sep 17 00:33:53.946377 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.946283    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:53.948781 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.948763    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4"]
Sep 17 00:33:53.948894 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.948855    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-59g6x"
Sep 17 00:33:53.949364 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.949346    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 00:33:53.950769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.950751    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 00:33:53.951953 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.951359    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gs4ts\""
Sep 17 00:33:53.951953 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.951035    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 00:33:53.951953 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.951083    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 00:33:53.951953 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.950777    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mt9m6\""
Sep 17 00:33:53.952426 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.952407    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"]
Sep 17 00:33:53.953559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.953532    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"]
Sep 17 00:33:53.954639 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:53.954619    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-59g6x"]
Sep 17 00:33:54.033457 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.033424    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6947d5777d-d7tnw"]
Sep 17 00:33:54.036420 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.036404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.038792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.038769    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 00:33:54.038926 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.038812    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 00:33:54.039568 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.039551    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 00:33:54.039680 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.039556    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 00:33:54.039680 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.039652    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-z6qft\""
Sep 17 00:33:54.039816 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.039797    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 00:33:54.039884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.039819    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 00:33:54.049663 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.049645    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6947d5777d-d7tnw"]
Sep 17 00:33:54.071768 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.071703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:54.071768 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.071761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf22z\" (UniqueName: \"kubernetes.io/projected/f552e56b-f89d-4163-93fb-36c17aeb2301-kube-api-access-wf22z\") pod \"network-check-source-6f8b768455-59g6x\" (UID: \"f552e56b-f89d-4163-93fb-36c17aeb2301\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-59g6x"
Sep 17 00:33:54.071945 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.071786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqcb\" (UniqueName: \"kubernetes.io/projected/2ff5e10e-ee60-4f10-af70-33604774eebf-kube-api-access-sxqcb\") pod \"service-ca-operator-cf7c776b7-92jss\" (UID: \"2ff5e10e-ee60-4f10-af70-33604774eebf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.071945 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.071866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnqg8\" (UniqueName: \"kubernetes.io/projected/de73b2a9-de5e-4be3-a40e-4a2bb8c943c7-kube-api-access-dnqg8\") pod \"volume-data-source-validator-86695b7cc7-zpnc4\" (UID: \"de73b2a9-de5e-4be3-a40e-4a2bb8c943c7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4"
Sep 17 00:33:54.071945 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.071915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ff5e10e-ee60-4f10-af70-33604774eebf-serving-cert\") pod \"service-ca-operator-cf7c776b7-92jss\" (UID: \"2ff5e10e-ee60-4f10-af70-33604774eebf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.072107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.072000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ff5e10e-ee60-4f10-af70-33604774eebf-config\") pod \"service-ca-operator-cf7c776b7-92jss\" (UID: \"2ff5e10e-ee60-4f10-af70-33604774eebf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.072107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.072026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/77a48136-1b49-4db8-8b78-a3270c01f775-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:54.072107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.072050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhb54\" (UniqueName: \"kubernetes.io/projected/77a48136-1b49-4db8-8b78-a3270c01f775-kube-api-access-nhb54\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:54.152409 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.152377    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:54.172993 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.172961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ff5e10e-ee60-4f10-af70-33604774eebf-serving-cert\") pod \"service-ca-operator-cf7c776b7-92jss\" (UID: \"2ff5e10e-ee60-4f10-af70-33604774eebf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.173119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ff5e10e-ee60-4f10-af70-33604774eebf-config\") pod \"service-ca-operator-cf7c776b7-92jss\" (UID: \"2ff5e10e-ee60-4f10-af70-33604774eebf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.173119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/77a48136-1b49-4db8-8b78-a3270c01f775-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:54.173119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:54.173290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmq6\" (UniqueName: \"kubernetes.io/projected/78ce94d3-caf0-42e3-85a0-728042777ffa-kube-api-access-fdmq6\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.173290 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.173224    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 00:33:54.173290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf22z\" (UniqueName: \"kubernetes.io/projected/f552e56b-f89d-4163-93fb-36c17aeb2301-kube-api-access-wf22z\") pod \"network-check-source-6f8b768455-59g6x\" (UID: \"f552e56b-f89d-4163-93fb-36c17aeb2301\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-59g6x"
Sep 17 00:33:54.173290 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.173290    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls podName:77a48136-1b49-4db8-8b78-a3270c01f775 nodeName:}" failed. No retries permitted until 2026-09-17 00:33:54.673272437 +0000 UTC m=+230.628868755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8dbdr" (UID: "77a48136-1b49-4db8-8b78-a3270c01f775") : secret "cluster-monitoring-operator-tls" not found
Sep 17 00:33:54.173476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqcb\" (UniqueName: \"kubernetes.io/projected/2ff5e10e-ee60-4f10-af70-33604774eebf-kube-api-access-sxqcb\") pod \"service-ca-operator-cf7c776b7-92jss\" (UID: \"2ff5e10e-ee60-4f10-af70-33604774eebf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.173476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173379    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.173476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhb54\" (UniqueName: \"kubernetes.io/projected/77a48136-1b49-4db8-8b78-a3270c01f775-kube-api-access-nhb54\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:54.173476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-default-certificate\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.173701 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnqg8\" (UniqueName: \"kubernetes.io/projected/de73b2a9-de5e-4be3-a40e-4a2bb8c943c7-kube-api-access-dnqg8\") pod \"volume-data-source-validator-86695b7cc7-zpnc4\" (UID: \"de73b2a9-de5e-4be3-a40e-4a2bb8c943c7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4"
Sep 17 00:33:54.173701 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173521    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.173701 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-stats-auth\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.173701 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173688    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ff5e10e-ee60-4f10-af70-33604774eebf-config\") pod \"service-ca-operator-cf7c776b7-92jss\" (UID: \"2ff5e10e-ee60-4f10-af70-33604774eebf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.173909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.173810    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/77a48136-1b49-4db8-8b78-a3270c01f775-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:54.175610 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.175584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ff5e10e-ee60-4f10-af70-33604774eebf-serving-cert\") pod \"service-ca-operator-cf7c776b7-92jss\" (UID: \"2ff5e10e-ee60-4f10-af70-33604774eebf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.183627 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.183561    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqcb\" (UniqueName: \"kubernetes.io/projected/2ff5e10e-ee60-4f10-af70-33604774eebf-kube-api-access-sxqcb\") pod \"service-ca-operator-cf7c776b7-92jss\" (UID: \"2ff5e10e-ee60-4f10-af70-33604774eebf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.183770 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.183752    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnqg8\" (UniqueName: \"kubernetes.io/projected/de73b2a9-de5e-4be3-a40e-4a2bb8c943c7-kube-api-access-dnqg8\") pod \"volume-data-source-validator-86695b7cc7-zpnc4\" (UID: \"de73b2a9-de5e-4be3-a40e-4a2bb8c943c7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4"
Sep 17 00:33:54.183940 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.183919    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhb54\" (UniqueName: \"kubernetes.io/projected/77a48136-1b49-4db8-8b78-a3270c01f775-kube-api-access-nhb54\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:54.183997 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.183937    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf22z\" (UniqueName: \"kubernetes.io/projected/f552e56b-f89d-4163-93fb-36c17aeb2301-kube-api-access-wf22z\") pod \"network-check-source-6f8b768455-59g6x\" (UID: \"f552e56b-f89d-4163-93fb-36c17aeb2301\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-59g6x"
Sep 17 00:33:54.247767 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.247733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4"
Sep 17 00:33:54.252281 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.252260    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:54.256175 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.256139    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"
Sep 17 00:33:54.268125 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.268101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-59g6x"
Sep 17 00:33:54.274282 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.274256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.274397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.274294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-default-certificate\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.274397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.274318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.274507 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.274404    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 00:33:54.274507 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.274438    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:33:54.774413206 +0000 UTC m=+230.730009525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : configmap references non-existent config key: service-ca.crt
Sep 17 00:33:54.274507 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.274466    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:33:54.774452544 +0000 UTC m=+230.730048868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : secret "router-metrics-certs-default" not found
Sep 17 00:33:54.274507 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.274480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-stats-auth\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.274730 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.274548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmq6\" (UniqueName: \"kubernetes.io/projected/78ce94d3-caf0-42e3-85a0-728042777ffa-kube-api-access-fdmq6\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.277024 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.277006    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-default-certificate\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.277313 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.277291    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-stats-auth\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.282407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.282382    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmq6\" (UniqueName: \"kubernetes.io/projected/78ce94d3-caf0-42e3-85a0-728042777ffa-kube-api-access-fdmq6\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.515998 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.515918    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4"]
Sep 17 00:33:54.519601 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:33:54.519572    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde73b2a9_de5e_4be3_a40e_4a2bb8c943c7.slice/crio-85ce780bed311f25d6a3407c764674c30b2ab542081bf8de097644af8eb79bfd WatchSource:0}: Error finding container 85ce780bed311f25d6a3407c764674c30b2ab542081bf8de097644af8eb79bfd: Status 404 returned error can't find the container with id 85ce780bed311f25d6a3407c764674c30b2ab542081bf8de097644af8eb79bfd
Sep 17 00:33:54.539870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.539845    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss"]
Sep 17 00:33:54.542787 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:33:54.542758    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff5e10e_ee60_4f10_af70_33604774eebf.slice/crio-5f4ecc2fb519c53c01bf83011ff3d759d91f0a12aad1ae4d8de0603947d11f7d WatchSource:0}: Error finding container 5f4ecc2fb519c53c01bf83011ff3d759d91f0a12aad1ae4d8de0603947d11f7d: Status 404 returned error can't find the container with id 5f4ecc2fb519c53c01bf83011ff3d759d91f0a12aad1ae4d8de0603947d11f7d
Sep 17 00:33:54.549112 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.549089    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-59g6x"]
Sep 17 00:33:54.551753 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:33:54.551728    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf552e56b_f89d_4163_93fb_36c17aeb2301.slice/crio-25e628cc252e9efb42ab7c7db1949fbf7619d230bcf6260e58d3127e95d5ee89 WatchSource:0}: Error finding container 25e628cc252e9efb42ab7c7db1949fbf7619d230bcf6260e58d3127e95d5ee89: Status 404 returned error can't find the container with id 25e628cc252e9efb42ab7c7db1949fbf7619d230bcf6260e58d3127e95d5ee89
Sep 17 00:33:54.678242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.678205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:54.678410 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.678364    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 00:33:54.678450 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.678430    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls podName:77a48136-1b49-4db8-8b78-a3270c01f775 nodeName:}" failed. No retries permitted until 2026-09-17 00:33:55.678413515 +0000 UTC m=+231.634009837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8dbdr" (UID: "77a48136-1b49-4db8-8b78-a3270c01f775") : secret "cluster-monitoring-operator-tls" not found
Sep 17 00:33:54.779280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.779250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.779439 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:54.779300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:54.779439 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.779400    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 00:33:54.779439 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.779415    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:33:55.779395946 +0000 UTC m=+231.734992265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : configmap references non-existent config key: service-ca.crt
Sep 17 00:33:54.779439 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:54.779442    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:33:55.779431053 +0000 UTC m=+231.735027372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : secret "router-metrics-certs-default" not found
Sep 17 00:33:55.277024 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:55.276989    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-59g6x" event={"ID":"f552e56b-f89d-4163-93fb-36c17aeb2301","Type":"ContainerStarted","Data":"ec00731bd95d8db14f6a21663a49eaec1bc92770f4b6c3b3d6e40437f96a2340"}
Sep 17 00:33:55.277496 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:55.277054    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-59g6x" event={"ID":"f552e56b-f89d-4163-93fb-36c17aeb2301","Type":"ContainerStarted","Data":"25e628cc252e9efb42ab7c7db1949fbf7619d230bcf6260e58d3127e95d5ee89"}
Sep 17 00:33:55.278523 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:55.278479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss" event={"ID":"2ff5e10e-ee60-4f10-af70-33604774eebf","Type":"ContainerStarted","Data":"5f4ecc2fb519c53c01bf83011ff3d759d91f0a12aad1ae4d8de0603947d11f7d"}
Sep 17 00:33:55.279983 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:55.279951    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4" event={"ID":"de73b2a9-de5e-4be3-a40e-4a2bb8c943c7","Type":"ContainerStarted","Data":"85ce780bed311f25d6a3407c764674c30b2ab542081bf8de097644af8eb79bfd"}
Sep 17 00:33:55.293342 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:55.293275    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-59g6x" podStartSLOduration=2.293256917 podStartE2EDuration="2.293256917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:33:55.292384764 +0000 UTC m=+231.247981107" watchObservedRunningTime="2026-09-17 00:33:55.293256917 +0000 UTC m=+231.248853260"
Sep 17 00:33:55.686630 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:55.686511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:55.686781 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:55.686682    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 00:33:55.686781 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:55.686748    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls podName:77a48136-1b49-4db8-8b78-a3270c01f775 nodeName:}" failed. No retries permitted until 2026-09-17 00:33:57.686729264 +0000 UTC m=+233.642325598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8dbdr" (UID: "77a48136-1b49-4db8-8b78-a3270c01f775") : secret "cluster-monitoring-operator-tls" not found
Sep 17 00:33:55.787429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:55.787391    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:55.787628 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:55.787537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:55.787628 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:55.787568    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 00:33:55.787740 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:55.787652    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:33:57.787629091 +0000 UTC m=+233.743225410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : secret "router-metrics-certs-default" not found
Sep 17 00:33:55.787740 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:55.787702    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:33:57.787681916 +0000 UTC m=+233.743278258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : configmap references non-existent config key: service-ca.crt
Sep 17 00:33:56.285575 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:56.285535    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4" event={"ID":"de73b2a9-de5e-4be3-a40e-4a2bb8c943c7","Type":"ContainerStarted","Data":"6f41847cc23f43300dfad0fe0564dd5d27a4a479c4bc7e0c216a13978b994658"}
Sep 17 00:33:56.300976 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:56.300922    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zpnc4" podStartSLOduration=1.921005805 podStartE2EDuration="3.300902936s" totalImagesPullingTime="1.379897131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:33:56.299892492 +0000 UTC m=+232.255488834" watchObservedRunningTime="2026-09-17 00:33:56.300902936 +0000 UTC m=+232.256499280"
Sep 17 00:33:56.380256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:56.380226    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:56.430157 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:56.430104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:57.290276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:57.290240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss" event={"ID":"2ff5e10e-ee60-4f10-af70-33604774eebf","Type":"ContainerStarted","Data":"ec8fe24272094d68de3440fd2ed1d07b94da03e10bc90e792827b809ef02f6f7"}
Sep 17 00:33:57.305797 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:57.305752    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss" podStartSLOduration=2.239586982 podStartE2EDuration="4.305734582s" totalImagesPullingTime="2.0661476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:33:57.304751115 +0000 UTC m=+233.260347456" watchObservedRunningTime="2026-09-17 00:33:57.305734582 +0000 UTC m=+233.261330924"
Sep 17 00:33:57.702978 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:57.702892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:33:57.703129 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:57.703005    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 00:33:57.703129 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:57.703088    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls podName:77a48136-1b49-4db8-8b78-a3270c01f775 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:01.703067927 +0000 UTC m=+237.658664259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8dbdr" (UID: "77a48136-1b49-4db8-8b78-a3270c01f775") : secret "cluster-monitoring-operator-tls" not found
Sep 17 00:33:57.804059 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:57.804018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:57.804239 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:57.804132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:33:57.804239 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:57.804201    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 00:33:57.804318 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:57.804260    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:34:01.804245468 +0000 UTC m=+237.759841787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : secret "router-metrics-certs-default" not found
Sep 17 00:33:57.804318 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:57.804294    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:34:01.80428808 +0000 UTC m=+237.759884398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : configmap references non-existent config key: service-ca.crt
Sep 17 00:33:57.838266 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:57.838231    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:57.893990 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:57.893961    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:58.528848 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.528808    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:58.589206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.589180    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:58.808069 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.807981    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vsc4b"]
Sep 17 00:33:58.811108 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.811085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:58.813360 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.813342    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 00:33:58.813620 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.813599    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 00:33:58.813691 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.813682    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 00:33:58.813751 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.813723    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 00:33:58.814774 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.814757    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4ctq5\""
Sep 17 00:33:58.817127 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.817104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:58.820499 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.820479    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vsc4b"]
Sep 17 00:33:58.875413 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.875380    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:58.911838 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.911804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ced1b4b1-4a7b-4365-8927-bffa7b60740d-data-volume\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:58.912030 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.911856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ced1b4b1-4a7b-4365-8927-bffa7b60740d-crio-socket\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:58.912030 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.911882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:58.912030 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.911908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:58.912030 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:58.911967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfw6\" (UniqueName: \"kubernetes.io/projected/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-api-access-xnfw6\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.012373 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.012336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ced1b4b1-4a7b-4365-8927-bffa7b60740d-data-volume\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.012373 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.012376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ced1b4b1-4a7b-4365-8927-bffa7b60740d-crio-socket\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.012513 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.012399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.012513 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.012419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.012513 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.012474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfw6\" (UniqueName: \"kubernetes.io/projected/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-api-access-xnfw6\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.012513 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.012479    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ced1b4b1-4a7b-4365-8927-bffa7b60740d-crio-socket\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.012513 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:59.012491    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 00:33:59.012686 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:59.012567    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:33:59.012686 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:59.012581    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:33:59.512560775 +0000 UTC m=+235.468157102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : configmap "kube-rbac-proxy" not found
Sep 17 00:33:59.012686 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:59.012608    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:33:59.512596947 +0000 UTC m=+235.468193266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:33:59.012686 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.012668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ced1b4b1-4a7b-4365-8927-bffa7b60740d-data-volume\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.021118 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.021094    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfw6\" (UniqueName: \"kubernetes.io/projected/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-api-access-xnfw6\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.516200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.516138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.516404 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:59.516258    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:33:59.516404 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.516274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:33:59.516404 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:59.516349    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:34:00.516328696 +0000 UTC m=+236.471925015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:33:59.516404 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:59.516355    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 00:33:59.516404 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:33:59.516395    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:34:00.516382148 +0000 UTC m=+236.471978467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : configmap "kube-rbac-proxy" not found
Sep 17 00:33:59.655665 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.655630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:59.756940 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.756910    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:33:59.857600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:33:59.857566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:00.126747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.126671    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-tlmrq"]
Sep 17 00:34:00.129775 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.129761    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.132462 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.132441    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 00:34:00.132582 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.132441    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rfw78\""
Sep 17 00:34:00.132582 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.132477    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 00:34:00.133864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.133839    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 00:34:00.133962 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.133894    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 00:34:00.137328 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.137301    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-tlmrq"]
Sep 17 00:34:00.156997 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.156969    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:00.207846 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.207816    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:00.221522 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.221487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e4d3c302-90e2-412e-b606-8f4bcde5d1e3-signing-key\") pod \"service-ca-568869fdc5-tlmrq\" (UID: \"e4d3c302-90e2-412e-b606-8f4bcde5d1e3\") " pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.221663 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.221608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzzz9\" (UniqueName: \"kubernetes.io/projected/e4d3c302-90e2-412e-b606-8f4bcde5d1e3-kube-api-access-vzzz9\") pod \"service-ca-568869fdc5-tlmrq\" (UID: \"e4d3c302-90e2-412e-b606-8f4bcde5d1e3\") " pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.221727 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.221659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e4d3c302-90e2-412e-b606-8f4bcde5d1e3-signing-cabundle\") pod \"service-ca-568869fdc5-tlmrq\" (UID: \"e4d3c302-90e2-412e-b606-8f4bcde5d1e3\") " pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.322414 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.322380    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzzz9\" (UniqueName: \"kubernetes.io/projected/e4d3c302-90e2-412e-b606-8f4bcde5d1e3-kube-api-access-vzzz9\") pod \"service-ca-568869fdc5-tlmrq\" (UID: \"e4d3c302-90e2-412e-b606-8f4bcde5d1e3\") " pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.322600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.322427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e4d3c302-90e2-412e-b606-8f4bcde5d1e3-signing-cabundle\") pod \"service-ca-568869fdc5-tlmrq\" (UID: \"e4d3c302-90e2-412e-b606-8f4bcde5d1e3\") " pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.322733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.322715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e4d3c302-90e2-412e-b606-8f4bcde5d1e3-signing-key\") pod \"service-ca-568869fdc5-tlmrq\" (UID: \"e4d3c302-90e2-412e-b606-8f4bcde5d1e3\") " pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.323032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.323016    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e4d3c302-90e2-412e-b606-8f4bcde5d1e3-signing-cabundle\") pod \"service-ca-568869fdc5-tlmrq\" (UID: \"e4d3c302-90e2-412e-b606-8f4bcde5d1e3\") " pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.325190 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.325173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e4d3c302-90e2-412e-b606-8f4bcde5d1e3-signing-key\") pod \"service-ca-568869fdc5-tlmrq\" (UID: \"e4d3c302-90e2-412e-b606-8f4bcde5d1e3\") " pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.330704 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.330680    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzzz9\" (UniqueName: \"kubernetes.io/projected/e4d3c302-90e2-412e-b606-8f4bcde5d1e3-kube-api-access-vzzz9\") pod \"service-ca-568869fdc5-tlmrq\" (UID: \"e4d3c302-90e2-412e-b606-8f4bcde5d1e3\") " pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.443898 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.443802    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-tlmrq"
Sep 17 00:34:00.524674 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.524641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:00.524674 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.524677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:00.524845 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:00.524765    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 00:34:00.524845 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:00.524780    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:34:00.524845 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:00.524830    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:34:02.524815544 +0000 UTC m=+238.480411868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : configmap "kube-rbac-proxy" not found
Sep 17 00:34:00.524972 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:00.524903    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:34:02.524878243 +0000 UTC m=+238.480474572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:34:00.716319 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.716227    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-tlmrq"]
Sep 17 00:34:00.720648 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:00.720621    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d3c302_90e2_412e_b606_8f4bcde5d1e3.slice/crio-366d0bee367aecde3a4b31dcaa5fc139092d1a1c77bd4fea15eafa922a7296e7 WatchSource:0}: Error finding container 366d0bee367aecde3a4b31dcaa5fc139092d1a1c77bd4fea15eafa922a7296e7: Status 404 returned error can't find the container with id 366d0bee367aecde3a4b31dcaa5fc139092d1a1c77bd4fea15eafa922a7296e7
Sep 17 00:34:00.968330 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.968251    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vhmzk_c99ec850-d4b6-4541-90a4-d158112c2570/dns-node-resolver/0.log"
Sep 17 00:34:00.992853 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:00.992824    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:01.304639 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:01.304608    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-tlmrq" event={"ID":"e4d3c302-90e2-412e-b606-8f4bcde5d1e3","Type":"ContainerStarted","Data":"1fc661aca794aa3d1266608099a2b489f5d210f96f4df7f78636331da7fbf61e"}
Sep 17 00:34:01.304801 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:01.304653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-tlmrq" event={"ID":"e4d3c302-90e2-412e-b606-8f4bcde5d1e3","Type":"ContainerStarted","Data":"366d0bee367aecde3a4b31dcaa5fc139092d1a1c77bd4fea15eafa922a7296e7"}
Sep 17 00:34:01.320595 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:01.320543    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-tlmrq" podStartSLOduration=1.320528534 podStartE2EDuration="1.320528534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:01.319786871 +0000 UTC m=+237.275383213" watchObservedRunningTime="2026-09-17 00:34:01.320528534 +0000 UTC m=+237.276124926"
Sep 17 00:34:01.568446 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:01.568368    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cf4c4_42d25648-164b-4ad8-9d6e-d38f1f3301e4/node-ca/0.log"
Sep 17 00:34:01.737686 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:01.737652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:34:01.738084 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:01.737816    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 00:34:01.738084 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:01.737881    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls podName:77a48136-1b49-4db8-8b78-a3270c01f775 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:09.737865806 +0000 UTC m=+245.693462125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8dbdr" (UID: "77a48136-1b49-4db8-8b78-a3270c01f775") : secret "cluster-monitoring-operator-tls" not found
Sep 17 00:34:01.838855 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:01.838769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:01.839008 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:01.838886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:01.839008 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:01.838924    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 00:34:01.839008 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:01.838992    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:34:09.838975578 +0000 UTC m=+245.794571911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : configmap references non-existent config key: service-ca.crt
Sep 17 00:34:01.839130 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:01.839012    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs podName:78ce94d3-caf0-42e3-85a0-728042777ffa nodeName:}" failed. No retries permitted until 2026-09-17 00:34:09.839002499 +0000 UTC m=+245.794598817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs") pod "router-default-6947d5777d-d7tnw" (UID: "78ce94d3-caf0-42e3-85a0-728042777ffa") : secret "router-metrics-certs-default" not found
Sep 17 00:34:02.343328 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:02.343290    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:02.394875 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:02.394837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:02.513020 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:02.512983    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:02.544171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:02.544112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:02.544372 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:02.544177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:02.544372 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:02.544250    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 00:34:02.544372 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:02.544321    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:34:06.544299885 +0000 UTC m=+242.499896218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : configmap "kube-rbac-proxy" not found
Sep 17 00:34:02.544579 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:02.544377    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:34:02.544579 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:02.544429    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:34:06.544411849 +0000 UTC m=+242.500008185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:34:02.614387 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:02.614299    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:02.715031 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:02.714997    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:02.768074 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:02.768027    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x6lxh_f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc/serve-healthcheck-canary/0.log"
Sep 17 00:34:02.814443 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:02.814408    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:03.037787 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:03.037749    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:03.278616 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:03.278569    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:04.041371 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:04.041335    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:04.437000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:04.436915    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:04.638855 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:04.638822    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:04.875362 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:04.875318    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:05.238592 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:05.238489    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:05.636052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:05.636019    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:06.031443 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:06.031415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:06.080242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:06.080204    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:06.230199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:06.230137    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:06.582646 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:06.582611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:06.583054 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:06.582654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:06.583054 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:06.582739    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 00:34:06.583054 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:06.582800    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:34:14.582783886 +0000 UTC m=+250.538380205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : configmap "kube-rbac-proxy" not found
Sep 17 00:34:06.583054 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:06.582856    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:34:06.583054 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:06.582923    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls podName:ced1b4b1-4a7b-4365-8927-bffa7b60740d nodeName:}" failed. No retries permitted until 2026-09-17 00:34:14.582905529 +0000 UTC m=+250.538501856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsc4b" (UID: "ced1b4b1-4a7b-4365-8927-bffa7b60740d") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:34:06.634694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:06.634658    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:07.043334 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:07.043303    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:07.436017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:07.435927    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:07.836293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:07.836256    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:08.038733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:08.038695    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:08.235643 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:08.235561    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:08.597632 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:08.597597    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:08.849643 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:08.849565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:09.237161 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.237057    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:09.437421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.437389    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:09.806681 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.806642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:34:09.809275 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.809245    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/77a48136-1b49-4db8-8b78-a3270c01f775-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8dbdr\" (UID: \"77a48136-1b49-4db8-8b78-a3270c01f775\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:34:09.838683 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.838656    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:09.864636 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.864609    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mt9m6\""
Sep 17 00:34:09.873130 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.873105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"
Sep 17 00:34:09.907765 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.907731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:09.907925 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.907825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:09.908375 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.908352    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ce94d3-caf0-42e3-85a0-728042777ffa-service-ca-bundle\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:09.910217 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.910199    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ce94d3-caf0-42e3-85a0-728042777ffa-metrics-certs\") pod \"router-default-6947d5777d-d7tnw\" (UID: \"78ce94d3-caf0-42e3-85a0-728042777ffa\") " pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:09.947916 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.947892    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-z6qft\""
Sep 17 00:34:09.956483 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:09.956448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:10.035081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.035051    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:10.148829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.148708    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr"]
Sep 17 00:34:10.151118 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:10.151089    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a48136_1b49_4db8_8b78_a3270c01f775.slice/crio-333771bfff34f7eac3f558a538d58fc6a756fcb3259169ac8e90fe88cc31b958 WatchSource:0}: Error finding container 333771bfff34f7eac3f558a538d58fc6a756fcb3259169ac8e90fe88cc31b958: Status 404 returned error can't find the container with id 333771bfff34f7eac3f558a538d58fc6a756fcb3259169ac8e90fe88cc31b958
Sep 17 00:34:10.230772 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.230727    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6947d5777d-d7tnw"]
Sep 17 00:34:10.234059 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:10.234033    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ce94d3_caf0_42e3_85a0_728042777ffa.slice/crio-41ef1874b171b97bffdf7674bef49182006a1272ec565e865151e872ca0559aa WatchSource:0}: Error finding container 41ef1874b171b97bffdf7674bef49182006a1272ec565e865151e872ca0559aa: Status 404 returned error can't find the container with id 41ef1874b171b97bffdf7674bef49182006a1272ec565e865151e872ca0559aa
Sep 17 00:34:10.339083 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.339051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6947d5777d-d7tnw" event={"ID":"78ce94d3-caf0-42e3-85a0-728042777ffa","Type":"ContainerStarted","Data":"5c067db308106134eee132ce7a6b59a66b53e50cc4510f7023ddc522b924edfe"}
Sep 17 00:34:10.339267 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.339103    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6947d5777d-d7tnw" event={"ID":"78ce94d3-caf0-42e3-85a0-728042777ffa","Type":"ContainerStarted","Data":"41ef1874b171b97bffdf7674bef49182006a1272ec565e865151e872ca0559aa"}
Sep 17 00:34:10.340427 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.340401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr" event={"ID":"77a48136-1b49-4db8-8b78-a3270c01f775","Type":"ContainerStarted","Data":"333771bfff34f7eac3f558a538d58fc6a756fcb3259169ac8e90fe88cc31b958"}
Sep 17 00:34:10.356420 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.356370    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6947d5777d-d7tnw" podStartSLOduration=16.356351864 podStartE2EDuration="16.356351864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:10.354989097 +0000 UTC m=+246.310585439" watchObservedRunningTime="2026-09-17 00:34:10.356351864 +0000 UTC m=+246.311948208"
Sep 17 00:34:10.636582 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.636539    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:10.957102 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.957023    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:10.959872 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:10.959851    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:11.033198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:11.033138    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:11.134833 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:11.134786    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:11.239251 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:11.239165    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:11.343728 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:11.343695    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:11.345093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:11.345075    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6947d5777d-d7tnw"
Sep 17 00:34:11.638188 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:11.638139    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:12.238598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:12.238568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:12.339591 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:12.339554    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:12.348760 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:12.348729    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr" event={"ID":"77a48136-1b49-4db8-8b78-a3270c01f775","Type":"ContainerStarted","Data":"061f963fc17d9d1a7b665db0e5cd6b14f1bc001c421acf53128e46289508be4b"}
Sep 17 00:34:12.365435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:12.365380    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8dbdr" podStartSLOduration=17.507990166 podStartE2EDuration="19.365357439s" totalImagesPullingTime="1.857367273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:12.364012569 +0000 UTC m=+248.319608911" watchObservedRunningTime="2026-09-17 00:34:12.365357439 +0000 UTC m=+248.320953782"
Sep 17 00:34:12.471081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:12.471049    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:12.634185 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:12.634054    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:12.865136 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:12.865104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:13.041847 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:13.041819    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:13.434731 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:13.434644    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:14.649733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:14.649693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:14.649733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:14.649735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:14.650254 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:14.650227    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ced1b4b1-4a7b-4365-8927-bffa7b60740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:14.652189 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:14.652165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ced1b4b1-4a7b-4365-8927-bffa7b60740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsc4b\" (UID: \"ced1b4b1-4a7b-4365-8927-bffa7b60740d\") " pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:14.722633 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:14.722593    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4ctq5\""
Sep 17 00:34:14.730837 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:14.730813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vsc4b"
Sep 17 00:34:15.013479 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:15.013407    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vsc4b"]
Sep 17 00:34:15.016580 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:15.016555    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced1b4b1_4a7b_4365_8927_bffa7b60740d.slice/crio-250df4e73bc9a362e2f697d978fc1ccd8a8f38f38a891fce7d5bc2e6efdd010f WatchSource:0}: Error finding container 250df4e73bc9a362e2f697d978fc1ccd8a8f38f38a891fce7d5bc2e6efdd010f: Status 404 returned error can't find the container with id 250df4e73bc9a362e2f697d978fc1ccd8a8f38f38a891fce7d5bc2e6efdd010f
Sep 17 00:34:15.360373 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:15.360340    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vsc4b" event={"ID":"ced1b4b1-4a7b-4365-8927-bffa7b60740d","Type":"ContainerStarted","Data":"9f1c2f8bfd02cba6e817e3214dc5c159c29f285900352663280e3337e0cf13bc"}
Sep 17 00:34:15.360550 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:15.360398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vsc4b" event={"ID":"ced1b4b1-4a7b-4365-8927-bffa7b60740d","Type":"ContainerStarted","Data":"250df4e73bc9a362e2f697d978fc1ccd8a8f38f38a891fce7d5bc2e6efdd010f"}
Sep 17 00:34:16.362276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:16.362239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:34:16.365248 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:16.365220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06683d51-91e1-4e99-a67d-75d8a715b5ab-metrics-certs\") pod \"network-metrics-daemon-w9nj2\" (UID: \"06683d51-91e1-4e99-a67d-75d8a715b5ab\") " pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:34:16.366360 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:16.366339    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vsc4b" event={"ID":"ced1b4b1-4a7b-4365-8927-bffa7b60740d","Type":"ContainerStarted","Data":"36b1451ca4f87f0ac24164d8bf727ff2c7dfb6cf50c3d89ab8464924595aad3a"}
Sep 17 00:34:16.385153 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:16.385128    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plv5t\""
Sep 17 00:34:16.393197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:16.393177    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w9nj2"
Sep 17 00:34:16.663309 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:16.663269    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w9nj2"]
Sep 17 00:34:16.665596 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:16.665567    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06683d51_91e1_4e99_a67d_75d8a715b5ab.slice/crio-ab273d8316de457779ceeb5ecdde3ebb385014ec1d0185076d70603f6a061108 WatchSource:0}: Error finding container ab273d8316de457779ceeb5ecdde3ebb385014ec1d0185076d70603f6a061108: Status 404 returned error can't find the container with id ab273d8316de457779ceeb5ecdde3ebb385014ec1d0185076d70603f6a061108
Sep 17 00:34:17.370886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:17.370862    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w9nj2" event={"ID":"06683d51-91e1-4e99-a67d-75d8a715b5ab","Type":"ContainerStarted","Data":"ab273d8316de457779ceeb5ecdde3ebb385014ec1d0185076d70603f6a061108"}
Sep 17 00:34:18.378096 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:18.378059    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vsc4b" event={"ID":"ced1b4b1-4a7b-4365-8927-bffa7b60740d","Type":"ContainerStarted","Data":"b6e5a8630e5fc0e14931b0858c61faf6e25d3d82340bc26f7e4cf6ba77ed421b"}
Sep 17 00:34:18.379752 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:18.379725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w9nj2" event={"ID":"06683d51-91e1-4e99-a67d-75d8a715b5ab","Type":"ContainerStarted","Data":"f696bc5e1c6fa6f2802d43ae3d0d4025aca1f958556bad3f1aa8a9a78233a579"}
Sep 17 00:34:18.379867 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:18.379776    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w9nj2" event={"ID":"06683d51-91e1-4e99-a67d-75d8a715b5ab","Type":"ContainerStarted","Data":"f3eec58b4eb9dfd3dd518607f7d28fcbc87a67c537afbbe8bef88d915040f124"}
Sep 17 00:34:18.395070 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:18.395026    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vsc4b" podStartSLOduration=18.157206539 podStartE2EDuration="20.395012028s" totalImagesPullingTime="2.237805489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:33:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:18.393428086 +0000 UTC m=+254.349024431" watchObservedRunningTime="2026-09-17 00:34:18.395012028 +0000 UTC m=+254.350608368"
Sep 17 00:34:18.407323 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:18.407277    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w9nj2" podStartSLOduration=252.905117395 podStartE2EDuration="4m14.407262967s" totalImagesPullingTime="1.502145572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:18.406298471 +0000 UTC m=+254.361894814" watchObservedRunningTime="2026-09-17 00:34:18.407262967 +0000 UTC m=+254.362859307"
Sep 17 00:34:18.778331 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:18.778297    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:20.442289 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:20.442258    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:20.680722 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:20.680694    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:20.863027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:20.862992    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:21.681833 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:21.681804    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:21.782493 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:21.782465    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:23.142615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:23.142588    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:23.234338 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:23.234304    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:23.331007 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:23.330979    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:26.498921 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:26.498844    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:26.558930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:26.558887    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:26.827623 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:26.827588    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:28.855297 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:28.855236    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:29.698059 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.698028    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bfbcb9b49-p5rfc"]
Sep 17 00:34:29.700643 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.700625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.703416 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.703384    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 00:34:29.703416 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.703396    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 00:34:29.703605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.703452    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 00:34:29.704119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.704100    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 00:34:29.704237 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.704127    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-px6mv\""
Sep 17 00:34:29.704237 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.704130    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 00:34:29.704407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.704389    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 00:34:29.704474 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.704418    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 00:34:29.710319 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.710291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bfbcb9b49-p5rfc"]
Sep 17 00:34:29.760779 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.760748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-serving-cert\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.760779 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.760779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-config\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.760967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.760803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759zc\" (UniqueName: \"kubernetes.io/projected/434bf0d4-94c9-4c13-bc41-7f892a841e8e-kube-api-access-759zc\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.760967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.760847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-oauth-serving-cert\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.760967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.760876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-service-ca\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.760967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.760892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-oauth-config\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.861792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.861741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759zc\" (UniqueName: \"kubernetes.io/projected/434bf0d4-94c9-4c13-bc41-7f892a841e8e-kube-api-access-759zc\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.862335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.861805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-oauth-serving-cert\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.862335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.861834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-service-ca\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.862335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.861853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-oauth-config\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.862335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.861965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-serving-cert\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.862335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.861990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-config\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.862722 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.862694    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-oauth-serving-cert\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.862792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.862697    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-service-ca\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.862792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.862706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-config\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.864509 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.864489    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-oauth-config\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.864589 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.864562    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-serving-cert\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:29.868984 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:29.868959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-759zc\" (UniqueName: \"kubernetes.io/projected/434bf0d4-94c9-4c13-bc41-7f892a841e8e-kube-api-access-759zc\") pod \"console-6bfbcb9b49-p5rfc\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") " pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:30.010466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:30.010364    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:30.280603 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:30.280570    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bfbcb9b49-p5rfc"]
Sep 17 00:34:30.283551 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:30.283528    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434bf0d4_94c9_4c13_bc41_7f892a841e8e.slice/crio-b3bb4d4de76ef2e48cdd965aebc1f5924c1580f63fd62164145824af462be285 WatchSource:0}: Error finding container b3bb4d4de76ef2e48cdd965aebc1f5924c1580f63fd62164145824af462be285: Status 404 returned error can't find the container with id b3bb4d4de76ef2e48cdd965aebc1f5924c1580f63fd62164145824af462be285
Sep 17 00:34:30.421318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:30.421277    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfbcb9b49-p5rfc" event={"ID":"434bf0d4-94c9-4c13-bc41-7f892a841e8e","Type":"ContainerStarted","Data":"b3bb4d4de76ef2e48cdd965aebc1f5924c1580f63fd62164145824af462be285"}
Sep 17 00:34:31.637551 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.637486    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:31.849683 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.849615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:31.888575 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.888493    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gbhrp"]
Sep 17 00:34:31.892300 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.892277    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.894760 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.894738    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 00:34:31.895092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.895073    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 00:34:31.895317 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.895295    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s8d7w\""
Sep 17 00:34:31.895434 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.895411    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 00:34:31.895870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.895856    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 00:34:31.937043 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.936954    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:31.980357 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.980310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-tls\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.980357 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.980357    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.980616 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.980392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-accelerators-collector-config\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.980616 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.980423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-textfile\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.980616 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.980461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/502e8041-e346-4b4c-a6ee-8d39143b4003-root\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.980616 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.980536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-wtmp\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.980616 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.980569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/502e8041-e346-4b4c-a6ee-8d39143b4003-sys\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.980616 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.980594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/502e8041-e346-4b4c-a6ee-8d39143b4003-metrics-client-ca\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.980964 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.980622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78h7x\" (UniqueName: \"kubernetes.io/projected/502e8041-e346-4b4c-a6ee-8d39143b4003-kube-api-access-78h7x\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:31.992031 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:31.992000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:32.081903 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.081844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-tls\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.081903 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.081888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.081918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-accelerators-collector-config\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.081947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-textfile\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.081982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/502e8041-e346-4b4c-a6ee-8d39143b4003-root\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.082015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-wtmp\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.082034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/502e8041-e346-4b4c-a6ee-8d39143b4003-sys\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.082055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/502e8041-e346-4b4c-a6ee-8d39143b4003-metrics-client-ca\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.082076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78h7x\" (UniqueName: \"kubernetes.io/projected/502e8041-e346-4b4c-a6ee-8d39143b4003-kube-api-access-78h7x\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082630 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.082552    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/502e8041-e346-4b4c-a6ee-8d39143b4003-root\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.082678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-wtmp\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.082925 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.082896    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/502e8041-e346-4b4c-a6ee-8d39143b4003-sys\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.084227 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.083561    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-accelerators-collector-config\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.084227 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.083986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/502e8041-e346-4b4c-a6ee-8d39143b4003-metrics-client-ca\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.084227 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.084193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-textfile\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.085687 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.085636    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-tls\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.085687 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.085679    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/502e8041-e346-4b4c-a6ee-8d39143b4003-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.091121 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.091097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78h7x\" (UniqueName: \"kubernetes.io/projected/502e8041-e346-4b4c-a6ee-8d39143b4003-kube-api-access-78h7x\") pod \"node-exporter-gbhrp\" (UID: \"502e8041-e346-4b4c-a6ee-8d39143b4003\") " pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.123074 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.123043    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:32.205496 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.205393    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gbhrp"
Sep 17 00:34:32.236894 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.236863    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:32.321127 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.321089    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:32.677946 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.677906    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:32.851515 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.851446    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:32.919152 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:32.919094    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:33.055865 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:33.055836    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502e8041_e346_4b4c_a6ee_8d39143b4003.slice/crio-b48174b73387b9cd1abba0c46bcf70427a68c90d3eba901788971d55a8c91e50 WatchSource:0}: Error finding container b48174b73387b9cd1abba0c46bcf70427a68c90d3eba901788971d55a8c91e50: Status 404 returned error can't find the container with id b48174b73387b9cd1abba0c46bcf70427a68c90d3eba901788971d55a8c91e50
Sep 17 00:34:33.278805 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:33.278762    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:33.435372 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:33.435276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfbcb9b49-p5rfc" event={"ID":"434bf0d4-94c9-4c13-bc41-7f892a841e8e","Type":"ContainerStarted","Data":"19e1b4d1d885e17454bdcf062bf923f281369ab2144be9d8b77260aefe5d9245"}
Sep 17 00:34:33.436686 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:33.436662    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gbhrp" event={"ID":"502e8041-e346-4b4c-a6ee-8d39143b4003","Type":"ContainerStarted","Data":"b48174b73387b9cd1abba0c46bcf70427a68c90d3eba901788971d55a8c91e50"}
Sep 17 00:34:33.453228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:33.453187    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bfbcb9b49-p5rfc" podStartSLOduration=1.6385407220000001 podStartE2EDuration="4.453171378s" totalImagesPullingTime="2.814630656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:33.452106689 +0000 UTC m=+269.407703044" watchObservedRunningTime="2026-09-17 00:34:33.453171378 +0000 UTC m=+269.408767721"
Sep 17 00:34:34.441652 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:34.441613    2576 generic.go:356] "Generic (PLEG): container finished" podID="502e8041-e346-4b4c-a6ee-8d39143b4003" containerID="aa30a74dedba397a6f9fbd3852008c134425bc765c6b049a4f869238ad8ce48d" exitCode=0
Sep 17 00:34:34.442061 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:34.441702    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gbhrp" event={"ID":"502e8041-e346-4b4c-a6ee-8d39143b4003","Type":"ContainerDied","Data":"aa30a74dedba397a6f9fbd3852008c134425bc765c6b049a4f869238ad8ce48d"}
Sep 17 00:34:34.779545 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:34.779516    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:34.880448 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:34.880420    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:35.088811 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:35.088730    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:35.189131 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:35.189097    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:35.290198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:35.290163    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:35.446992 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:35.446903    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gbhrp" event={"ID":"502e8041-e346-4b4c-a6ee-8d39143b4003","Type":"ContainerStarted","Data":"5610078010ccd0118e90da114556dc0d26ebab94eebd20942eaa3747a816f42f"}
Sep 17 00:34:35.446992 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:35.446957    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gbhrp" event={"ID":"502e8041-e346-4b4c-a6ee-8d39143b4003","Type":"ContainerStarted","Data":"97c7cb38907aba24971b3b104fa1a4f37b4102c670418364badddfc0ef0de42b"}
Sep 17 00:34:35.464970 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:35.464928    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gbhrp" podStartSLOduration=3.546421064 podStartE2EDuration="4.464915332s" totalImagesPullingTime="850.122577ms" totalInitContainerRuntime="68.371691ms" isStatefulPod=true podCreationTimestamp="2026-09-17 00:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:35.46388788 +0000 UTC m=+271.419484230" watchObservedRunningTime="2026-09-17 00:34:35.464915332 +0000 UTC m=+271.420511672"
Sep 17 00:34:36.645316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.645287    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:36.652600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.652572    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"]
Sep 17 00:34:36.656163 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.656131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:36.658397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.658373    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 00:34:36.658532 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.658488    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x4qxg\""
Sep 17 00:34:36.665064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.665045    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"]
Sep 17 00:34:36.699519 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.699488    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:36.722723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.722693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebad7518-ce1b-4ea3-8d53-82acba867e65-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7htk7\" (UID: \"ebad7518-ce1b-4ea3-8d53-82acba867e65\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:36.722855 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.722745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrcnz\" (UniqueName: \"kubernetes.io/projected/ebad7518-ce1b-4ea3-8d53-82acba867e65-kube-api-access-wrcnz\") pod \"monitoring-plugin-5d85778b9d-7htk7\" (UID: \"ebad7518-ce1b-4ea3-8d53-82acba867e65\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:36.816332 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.816293    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:36.823433 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.823404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebad7518-ce1b-4ea3-8d53-82acba867e65-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7htk7\" (UID: \"ebad7518-ce1b-4ea3-8d53-82acba867e65\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:36.823578 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.823462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrcnz\" (UniqueName: \"kubernetes.io/projected/ebad7518-ce1b-4ea3-8d53-82acba867e65-kube-api-access-wrcnz\") pod \"monitoring-plugin-5d85778b9d-7htk7\" (UID: \"ebad7518-ce1b-4ea3-8d53-82acba867e65\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:36.826029 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.826010    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ebad7518-ce1b-4ea3-8d53-82acba867e65-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7htk7\" (UID: \"ebad7518-ce1b-4ea3-8d53-82acba867e65\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:36.831637 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.831615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrcnz\" (UniqueName: \"kubernetes.io/projected/ebad7518-ce1b-4ea3-8d53-82acba867e65-kube-api-access-wrcnz\") pod \"monitoring-plugin-5d85778b9d-7htk7\" (UID: \"ebad7518-ce1b-4ea3-8d53-82acba867e65\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:36.967329 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:36.967213    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:37.235677 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:37.235591    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"]
Sep 17 00:34:37.238628 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:37.238596    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebad7518_ce1b_4ea3_8d53_82acba867e65.slice/crio-fdaa04e8a065eff72c1fdbb72740e3ebf4dc1105a3d9f55ec4c6dc2130935b95 WatchSource:0}: Error finding container fdaa04e8a065eff72c1fdbb72740e3ebf4dc1105a3d9f55ec4c6dc2130935b95: Status 404 returned error can't find the container with id fdaa04e8a065eff72c1fdbb72740e3ebf4dc1105a3d9f55ec4c6dc2130935b95
Sep 17 00:34:37.455638 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:37.455606    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7" event={"ID":"ebad7518-ce1b-4ea3-8d53-82acba867e65","Type":"ContainerStarted","Data":"fdaa04e8a065eff72c1fdbb72740e3ebf4dc1105a3d9f55ec4c6dc2130935b95"}
Sep 17 00:34:38.044603 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.044567    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 00:34:38.045705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.045679    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:38.048255 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.048239    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.052050 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.052027    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 00:34:38.052168 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.052104    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 00:34:38.052168 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.052117    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 00:34:38.052475 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.052456    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 00:34:38.052560 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.052488    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 00:34:38.052560 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.052497    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 00:34:38.052762 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.052748    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-74pvk\""
Sep 17 00:34:38.053053 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.053038    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 00:34:38.053130 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.053062    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 00:34:38.053130 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.053070    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 00:34:38.053278 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.053176    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 00:34:38.053497 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.053483    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 00:34:38.053725 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.053710    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 00:34:38.053796 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.053738    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-46lflvoio0kkg\""
Sep 17 00:34:38.053796 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.053785    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 00:34:38.054116 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.054099    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 00:34:38.055969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.055793    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 00:34:38.063244 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.063221    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 00:34:38.135379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-config\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135587 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135587 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135587 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135587 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135722 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135722 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135821 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135821 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd86n\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-kube-api-access-zd86n\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135917 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135917 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.135917 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.136063 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.136063 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.136063 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.135979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.136063 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.136002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.136063 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.136027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.136063 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.136052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.136415 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.136098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237392 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237392 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237754 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237754 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237754 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237754 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237754 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237981 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237981 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-config\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237981 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237981 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237878    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237981 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.237981 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.237927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.238292 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.238248    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.239871 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.239126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.239871 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.239813    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.240574 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.240550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.240723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.240705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.240854 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.240839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.240958 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.240945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd86n\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-kube-api-access-zd86n\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.241600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.241573    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.242063 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.242039    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.242167 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.242044    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.242402 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.242380    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.242618 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.242545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.242814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.242793    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.242940 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.240660    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.243199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.243179    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.243686 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.243660    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.243771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.243730    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-config\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.244067 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.244042    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.244346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.244326    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.244522 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.244499    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.245059 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.245038    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.246097 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.246072    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.246490 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.246469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.248677 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.248656    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd86n\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-kube-api-access-zd86n\") pod \"prometheus-k8s-0\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.358433 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.358336    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:38.809729 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:38.809618    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 00:34:38.811655 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:38.811628    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef44f77e_34bf_4c87_87d5_983882ef2035.slice/crio-020c4cdfc6ae041a9dfcf9820726e3914bd5bdf6d5a5c296ba73ba1f39613168 WatchSource:0}: Error finding container 020c4cdfc6ae041a9dfcf9820726e3914bd5bdf6d5a5c296ba73ba1f39613168: Status 404 returned error can't find the container with id 020c4cdfc6ae041a9dfcf9820726e3914bd5bdf6d5a5c296ba73ba1f39613168
Sep 17 00:34:39.465874 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:39.465833    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerStarted","Data":"020c4cdfc6ae041a9dfcf9820726e3914bd5bdf6d5a5c296ba73ba1f39613168"}
Sep 17 00:34:39.467667 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:39.467639    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7" event={"ID":"ebad7518-ce1b-4ea3-8d53-82acba867e65","Type":"ContainerStarted","Data":"f7f60ca62f8620ef50c077b22d209d2301825667c0fcc49db965bf297f326f50"}
Sep 17 00:34:39.467901 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:39.467878    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:39.473798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:39.473774    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7"
Sep 17 00:34:39.482420 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:39.482168    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7htk7" podStartSLOduration=2.141036955 podStartE2EDuration="3.482131958s" totalImagesPullingTime="1.341095003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:39.481869181 +0000 UTC m=+275.437465523" watchObservedRunningTime="2026-09-17 00:34:39.482131958 +0000 UTC m=+275.437728299"
Sep 17 00:34:40.010568 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:40.010534    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:40.010730 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:40.010613    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:34:40.011864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:40.011835    2576 patch_prober.go:28] interesting pod/console-6bfbcb9b49-p5rfc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body=
Sep 17 00:34:40.011996 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:40.011885    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6bfbcb9b49-p5rfc" podUID="434bf0d4-94c9-4c13-bc41-7f892a841e8e" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused"
Sep 17 00:34:40.475127 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:40.475096    2576 generic.go:356] "Generic (PLEG): container finished" podID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerID="18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9" exitCode=0
Sep 17 00:34:40.475613 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:40.475188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerDied","Data":"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"}
Sep 17 00:34:40.882740 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:40.882701    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:40.933572 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:40.933539    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:41.107202 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:41.107169    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:41.208242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:41.208138    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:42.908507 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:42.908476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:43.010184 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:43.010131    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:43.064814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:43.064769    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:44.495532 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:44.495499    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerStarted","Data":"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"}
Sep 17 00:34:44.495907 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:44.495572    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerStarted","Data":"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"}
Sep 17 00:34:45.093170 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:45.093104    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj" podUID="853a6578-a7f7-4ed0-9331-e9877dadf9ce"
Sep 17 00:34:45.093366 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:34:45.093104    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-r4k48" podUID="bc042542-1f09-45d0-b848-cbf7f11ff4e7"
Sep 17 00:34:45.110828 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:45.110797    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:45.210576 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:45.210540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:45.502666 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:45.502627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerStarted","Data":"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"}
Sep 17 00:34:45.502666 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:45.502652    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r4k48"
Sep 17 00:34:45.502666 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:45.502666    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:34:45.503196 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:45.502735    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerStarted","Data":"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"}
Sep 17 00:34:45.503196 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:45.502770    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerStarted","Data":"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"}
Sep 17 00:34:45.503196 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:45.502800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerStarted","Data":"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"}
Sep 17 00:34:45.531550 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:45.531496    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.938839094 podStartE2EDuration="7.531479451s" totalImagesPullingTime="5.573386562s" totalInitContainerRuntime="19.253795ms" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:45.528940151 +0000 UTC m=+281.484536495" watchObservedRunningTime="2026-09-17 00:34:45.531479451 +0000 UTC m=+281.487075792"
Sep 17 00:34:48.236498 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.236462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:34:48.236873 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.236549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:34:48.239115 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.239085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc042542-1f09-45d0-b848-cbf7f11ff4e7-metrics-tls\") pod \"dns-default-r4k48\" (UID: \"bc042542-1f09-45d0-b848-cbf7f11ff4e7\") " pod="openshift-dns/dns-default-r4k48"
Sep 17 00:34:48.239236 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.239202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"image-registry-5d46c8d77b-f7fpj\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") " pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:34:48.359352 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.359315    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:34:48.507045 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.506967    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n9svt\""
Sep 17 00:34:48.507045 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.506965    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cjf6n\""
Sep 17 00:34:48.515805 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.515771    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:34:48.516348 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.515837    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r4k48"
Sep 17 00:34:48.802133 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.802108    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r4k48"]
Sep 17 00:34:48.805313 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:48.805274    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc042542_1f09_45d0_b848_cbf7f11ff4e7.slice/crio-25e2597ecede0ffc6782f0ed96de7e51b60bcc8581802beea5a1fa94c2148215 WatchSource:0}: Error finding container 25e2597ecede0ffc6782f0ed96de7e51b60bcc8581802beea5a1fa94c2148215: Status 404 returned error can't find the container with id 25e2597ecede0ffc6782f0ed96de7e51b60bcc8581802beea5a1fa94c2148215
Sep 17 00:34:48.805500 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:48.805437    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d46c8d77b-f7fpj"]
Sep 17 00:34:48.808486 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:34:48.808465    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853a6578_a7f7_4ed0_9331_e9877dadf9ce.slice/crio-9faa04a4e77ca70c912cc9e420d045a981a1ce4ddc7b87f192865dff9b2dd045 WatchSource:0}: Error finding container 9faa04a4e77ca70c912cc9e420d045a981a1ce4ddc7b87f192865dff9b2dd045: Status 404 returned error can't find the container with id 9faa04a4e77ca70c912cc9e420d045a981a1ce4ddc7b87f192865dff9b2dd045
Sep 17 00:34:49.097789 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:49.097704    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:49.197782 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:49.197754    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:49.519612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:49.519518    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r4k48" event={"ID":"bc042542-1f09-45d0-b848-cbf7f11ff4e7","Type":"ContainerStarted","Data":"25e2597ecede0ffc6782f0ed96de7e51b60bcc8581802beea5a1fa94c2148215"}
Sep 17 00:34:49.521458 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:49.521412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj" event={"ID":"853a6578-a7f7-4ed0-9331-e9877dadf9ce","Type":"ContainerStarted","Data":"55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a"}
Sep 17 00:34:49.521571 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:49.521470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj" event={"ID":"853a6578-a7f7-4ed0-9331-e9877dadf9ce","Type":"ContainerStarted","Data":"9faa04a4e77ca70c912cc9e420d045a981a1ce4ddc7b87f192865dff9b2dd045"}
Sep 17 00:34:49.521571 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:49.521556    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:34:49.541500 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:49.541444    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj" podStartSLOduration=293.541427946 podStartE2EDuration="4m53.541427946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:49.538973316 +0000 UTC m=+285.494569657" watchObservedRunningTime="2026-09-17 00:34:49.541427946 +0000 UTC m=+285.497024287"
Sep 17 00:34:49.595569 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:49.595530    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:49.696866 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:49.696827    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:50.010876 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:50.010842    2576 patch_prober.go:28] interesting pod/console-6bfbcb9b49-p5rfc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body=
Sep 17 00:34:50.011073 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:50.010895    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6bfbcb9b49-p5rfc" podUID="434bf0d4-94c9-4c13-bc41-7f892a841e8e" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused"
Sep 17 00:34:51.530435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:51.530398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r4k48" event={"ID":"bc042542-1f09-45d0-b848-cbf7f11ff4e7","Type":"ContainerStarted","Data":"938737f7a497f27d14d22a26abada83f2cfaef76b0cbac52889741c81fbd5b6d"}
Sep 17 00:34:51.530832 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:51.530463    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r4k48" event={"ID":"bc042542-1f09-45d0-b848-cbf7f11ff4e7","Type":"ContainerStarted","Data":"fbe1428980512e7ec7da473984d29ee92313660cfe11e282a93668feaba16dac"}
Sep 17 00:34:51.530832 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:51.530512    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r4k48"
Sep 17 00:34:51.546152 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:51.546100    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r4k48" podStartSLOduration=251.638278168 podStartE2EDuration="4m13.546083442s" totalImagesPullingTime="1.907805274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:51.544923876 +0000 UTC m=+287.500520228" watchObservedRunningTime="2026-09-17 00:34:51.546083442 +0000 UTC m=+287.501679783"
Sep 17 00:34:55.061074 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:55.061038    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d46c8d77b-f7fpj"]
Sep 17 00:34:55.138166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:55.138114    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:55.238924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:55.238885    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:55.299021 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:55.298990    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:58.572126 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:58.572087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:58.618897 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:58.618864    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:59.494847 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:59.494811    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bfbcb9b49-p5rfc"]
Sep 17 00:34:59.588458 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:59.588430    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:59.639400 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:59.639367    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:34:59.695169 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:34:59.695115    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:01.536664 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:01.536635    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r4k48"
Sep 17 00:35:01.815244 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:01.815137    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:01.866523 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:01.866491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:02.896511 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:02.896477    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:02.997367 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:02.997339    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:04.035007 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:04.034972    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r4k48_bc042542-1f09-45d0-b848-cbf7f11ff4e7/dns/0.log"
Sep 17 00:35:04.234497 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:04.234472    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r4k48_bc042542-1f09-45d0-b848-cbf7f11ff4e7/kube-rbac-proxy/0.log"
Sep 17 00:35:04.468038 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:04.467968    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:35:04.468199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:04.468057    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:35:04.471875 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:04.471854    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 00:35:04.592123 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:04.592094    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:04.644010 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:04.643974    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:05.066846 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:05.066815    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:35:05.234866 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:05.234841    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vhmzk_c99ec850-d4b6-4541-90a4-d158112c2570/dns-node-resolver/0.log"
Sep 17 00:35:05.434588 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:05.434480    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6947d5777d-d7tnw_78ce94d3-caf0-42e3-85a0-728042777ffa/router/0.log"
Sep 17 00:35:06.034678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:06.034626    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x6lxh_f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc/serve-healthcheck-canary/0.log"
Sep 17 00:35:11.897690 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:11.897645    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:11.997775 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:11.997742    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:12.878504 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:12.878418    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:15.633590 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:15.633556    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:15.684233 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:15.684209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:19.593581 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:19.593545    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:19.694086 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:19.694053    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:20.080601 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.080541    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj" podUID="853a6578-a7f7-4ed0-9331-e9877dadf9ce" containerName="registry" containerID="cri-o://55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a" gracePeriod=30
Sep 17 00:35:20.334550 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.334485    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:35:20.401761 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.401732    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-trusted-ca\") pod \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") "
Sep 17 00:35:20.401930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.401775    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-image-registry-private-configuration\") pod \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") "
Sep 17 00:35:20.401930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.401805    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-certificates\") pod \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") "
Sep 17 00:35:20.401930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.401834    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-bound-sa-token\") pod \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") "
Sep 17 00:35:20.401930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.401862    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-installation-pull-secrets\") pod \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") "
Sep 17 00:35:20.401930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.401880    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/853a6578-a7f7-4ed0-9331-e9877dadf9ce-ca-trust-extracted\") pod \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") "
Sep 17 00:35:20.401930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.401915    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7kqk\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-kube-api-access-q7kqk\") pod \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") "
Sep 17 00:35:20.402243 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.401946    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") pod \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\" (UID: \"853a6578-a7f7-4ed0-9331-e9877dadf9ce\") "
Sep 17 00:35:20.402379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.402261    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "853a6578-a7f7-4ed0-9331-e9877dadf9ce" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:20.402648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.402585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "853a6578-a7f7-4ed0-9331-e9877dadf9ce" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:20.404555 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.404456    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-kube-api-access-q7kqk" (OuterVolumeSpecName: "kube-api-access-q7kqk") pod "853a6578-a7f7-4ed0-9331-e9877dadf9ce" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce"). InnerVolumeSpecName "kube-api-access-q7kqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:35:20.404697 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.404673    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "853a6578-a7f7-4ed0-9331-e9877dadf9ce" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:20.404794 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.404777    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "853a6578-a7f7-4ed0-9331-e9877dadf9ce" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:35:20.404794 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.404770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "853a6578-a7f7-4ed0-9331-e9877dadf9ce" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:20.404975 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.404958    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "853a6578-a7f7-4ed0-9331-e9877dadf9ce" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:35:20.410369 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.410344    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853a6578-a7f7-4ed0-9331-e9877dadf9ce-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "853a6578-a7f7-4ed0-9331-e9877dadf9ce" (UID: "853a6578-a7f7-4ed0-9331-e9877dadf9ce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:20.502885 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.502852    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-image-registry-private-configuration\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:20.502885 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.502881    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-certificates\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:20.502885 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.502891    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-bound-sa-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:20.503101 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.502900    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/853a6578-a7f7-4ed0-9331-e9877dadf9ce-installation-pull-secrets\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:20.503101 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.502932    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/853a6578-a7f7-4ed0-9331-e9877dadf9ce-ca-trust-extracted\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:20.503101 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.502941    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7kqk\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-kube-api-access-q7kqk\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:20.503101 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.502950    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853a6578-a7f7-4ed0-9331-e9877dadf9ce-registry-tls\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:20.503101 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.502957    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/853a6578-a7f7-4ed0-9331-e9877dadf9ce-trusted-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:20.650098 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.650060    2576 generic.go:356] "Generic (PLEG): container finished" podID="853a6578-a7f7-4ed0-9331-e9877dadf9ce" containerID="55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a" exitCode=0
Sep 17 00:35:20.650534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.650107    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj"
Sep 17 00:35:20.650534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.650107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj" event={"ID":"853a6578-a7f7-4ed0-9331-e9877dadf9ce","Type":"ContainerDied","Data":"55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a"}
Sep 17 00:35:20.650534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.650232    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d46c8d77b-f7fpj" event={"ID":"853a6578-a7f7-4ed0-9331-e9877dadf9ce","Type":"ContainerDied","Data":"9faa04a4e77ca70c912cc9e420d045a981a1ce4ddc7b87f192865dff9b2dd045"}
Sep 17 00:35:20.650534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.650279    2576 scope.go:117] "RemoveContainer" containerID="55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a"
Sep 17 00:35:20.658758 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.658741    2576 scope.go:117] "RemoveContainer" containerID="55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a"
Sep 17 00:35:20.659051 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:35:20.659023    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a\": container with ID starting with 55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a not found: ID does not exist" containerID="55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a"
Sep 17 00:35:20.659092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.659065    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a"} err="failed to get container status \"55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a\": rpc error: code = NotFound desc = could not find container \"55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a\": container with ID starting with 55ad438945838981fba5556a20d8c08a78a43563bbda180040f50f0673c2cc4a not found: ID does not exist"
Sep 17 00:35:20.668245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.668220    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d46c8d77b-f7fpj"]
Sep 17 00:35:20.671494 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.671471    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d46c8d77b-f7fpj"]
Sep 17 00:35:20.925516 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:20.925439    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:21.026097 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:21.026064    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:22.586496 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:22.586464    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="853a6578-a7f7-4ed0-9331-e9877dadf9ce" path="/var/lib/kubelet/pods/853a6578-a7f7-4ed0-9331-e9877dadf9ce/volumes"
Sep 17 00:35:24.514490 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.514454    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bfbcb9b49-p5rfc" podUID="434bf0d4-94c9-4c13-bc41-7f892a841e8e" containerName="console" containerID="cri-o://19e1b4d1d885e17454bdcf062bf923f281369ab2144be9d8b77260aefe5d9245" gracePeriod=15
Sep 17 00:35:24.667233 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.667196    2576 generic.go:356] "Generic (PLEG): container finished" podID="434bf0d4-94c9-4c13-bc41-7f892a841e8e" containerID="19e1b4d1d885e17454bdcf062bf923f281369ab2144be9d8b77260aefe5d9245" exitCode=0
Sep 17 00:35:24.667394 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.667268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfbcb9b49-p5rfc" event={"ID":"434bf0d4-94c9-4c13-bc41-7f892a841e8e","Type":"ContainerDied","Data":"19e1b4d1d885e17454bdcf062bf923f281369ab2144be9d8b77260aefe5d9245"}
Sep 17 00:35:24.716797 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.716758    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:24.748295 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.748269    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:35:24.818728 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.818686    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:24.838707 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.838678    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-oauth-serving-cert\") pod \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") "
Sep 17 00:35:24.838882 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.838729    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-759zc\" (UniqueName: \"kubernetes.io/projected/434bf0d4-94c9-4c13-bc41-7f892a841e8e-kube-api-access-759zc\") pod \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") "
Sep 17 00:35:24.838882 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.838747    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-oauth-config\") pod \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") "
Sep 17 00:35:24.838882 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.838765    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-service-ca\") pod \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") "
Sep 17 00:35:24.838882 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.838841    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-serving-cert\") pod \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") "
Sep 17 00:35:24.839079 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.838886    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-config\") pod \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\" (UID: \"434bf0d4-94c9-4c13-bc41-7f892a841e8e\") "
Sep 17 00:35:24.839248 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.839221    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-service-ca" (OuterVolumeSpecName: "service-ca") pod "434bf0d4-94c9-4c13-bc41-7f892a841e8e" (UID: "434bf0d4-94c9-4c13-bc41-7f892a841e8e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:24.839248 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.839229    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "434bf0d4-94c9-4c13-bc41-7f892a841e8e" (UID: "434bf0d4-94c9-4c13-bc41-7f892a841e8e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:24.839405 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.839298    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-config" (OuterVolumeSpecName: "console-config") pod "434bf0d4-94c9-4c13-bc41-7f892a841e8e" (UID: "434bf0d4-94c9-4c13-bc41-7f892a841e8e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:24.841379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.841358    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434bf0d4-94c9-4c13-bc41-7f892a841e8e-kube-api-access-759zc" (OuterVolumeSpecName: "kube-api-access-759zc") pod "434bf0d4-94c9-4c13-bc41-7f892a841e8e" (UID: "434bf0d4-94c9-4c13-bc41-7f892a841e8e"). InnerVolumeSpecName "kube-api-access-759zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:35:24.841466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.841391    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "434bf0d4-94c9-4c13-bc41-7f892a841e8e" (UID: "434bf0d4-94c9-4c13-bc41-7f892a841e8e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:24.841505 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.841451    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "434bf0d4-94c9-4c13-bc41-7f892a841e8e" (UID: "434bf0d4-94c9-4c13-bc41-7f892a841e8e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:24.939579 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.939539    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759zc\" (UniqueName: \"kubernetes.io/projected/434bf0d4-94c9-4c13-bc41-7f892a841e8e-kube-api-access-759zc\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:24.939579 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.939571    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-oauth-config\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:24.939579 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.939581    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:24.939579 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.939589    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-serving-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:24.939838 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.939597    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-console-config\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:24.939838 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:24.939606    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/434bf0d4-94c9-4c13-bc41-7f892a841e8e-oauth-serving-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:25.672768 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:25.672739    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfbcb9b49-p5rfc"
Sep 17 00:35:25.672768 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:25.672747    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfbcb9b49-p5rfc" event={"ID":"434bf0d4-94c9-4c13-bc41-7f892a841e8e","Type":"ContainerDied","Data":"b3bb4d4de76ef2e48cdd965aebc1f5924c1580f63fd62164145824af462be285"}
Sep 17 00:35:25.673324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:25.672799    2576 scope.go:117] "RemoveContainer" containerID="19e1b4d1d885e17454bdcf062bf923f281369ab2144be9d8b77260aefe5d9245"
Sep 17 00:35:25.694110 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:25.694074    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bfbcb9b49-p5rfc"]
Sep 17 00:35:25.699576 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:25.699554    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bfbcb9b49-p5rfc"]
Sep 17 00:35:26.586687 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:26.586655    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="434bf0d4-94c9-4c13-bc41-7f892a841e8e" path="/var/lib/kubelet/pods/434bf0d4-94c9-4c13-bc41-7f892a841e8e/volumes"
Sep 17 00:35:27.683605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:27.683526    2576 generic.go:356] "Generic (PLEG): container finished" podID="2ff5e10e-ee60-4f10-af70-33604774eebf" containerID="ec8fe24272094d68de3440fd2ed1d07b94da03e10bc90e792827b809ef02f6f7" exitCode=0
Sep 17 00:35:27.683948 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:27.683602    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss" event={"ID":"2ff5e10e-ee60-4f10-af70-33604774eebf","Type":"ContainerDied","Data":"ec8fe24272094d68de3440fd2ed1d07b94da03e10bc90e792827b809ef02f6f7"}
Sep 17 00:35:27.683948 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:27.683922    2576 scope.go:117] "RemoveContainer" containerID="ec8fe24272094d68de3440fd2ed1d07b94da03e10bc90e792827b809ef02f6f7"
Sep 17 00:35:28.688917 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:28.688883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-92jss" event={"ID":"2ff5e10e-ee60-4f10-af70-33604774eebf","Type":"ContainerStarted","Data":"602007e675abfe07fd71cf37cb1feec5d296ec1b3f2c4e14cdd0b79ee1121adc"}
Sep 17 00:35:34.593702 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:34.593672    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:34.649969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:34.649940    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:38.359276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:38.359227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:38.375403 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:38.375364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:38.742521 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:38.742447    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:42.013518 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:42.013484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:42.114202 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:42.114174    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:42.163430 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:42.163399    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:51.021302 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:51.021255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:51.648498 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:51.648443    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:54.417440 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:54.417402    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:56.415755 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.415596    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 00:35:56.416245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.415932    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="prometheus" containerID="cri-o://adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f" gracePeriod=600
Sep 17 00:35:56.416245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.415962    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy" containerID="cri-o://7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c" gracePeriod=600
Sep 17 00:35:56.416245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.416003    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy-web" containerID="cri-o://4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4" gracePeriod=600
Sep 17 00:35:56.416245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.416023    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e" gracePeriod=600
Sep 17 00:35:56.416245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.416053    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="config-reloader" containerID="cri-o://674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87" gracePeriod=600
Sep 17 00:35:56.416245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.415978    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="thanos-sidecar" containerID="cri-o://407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85" gracePeriod=600
Sep 17 00:35:56.657076 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.657053    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:56.786920 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.786896    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787050 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.786943    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-serving-certs-ca-bundle\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787050 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.786965    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-grpc-tls\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787050 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.786995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787050 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-1\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787050 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787030    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-config\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787051    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-0\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787076    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-thanos-prometheus-http-client-file\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787109    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-db\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787133    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd86n\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-kube-api-access-zd86n\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787192    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-kube-rbac-proxy\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787215    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-tls-assets\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787241    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-metrics-client-ca\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787272    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-metrics-client-certs\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-trusted-ca-bundle\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787336    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-tls\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787366    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-2\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787390    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-web-config\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787475    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-config-out\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.787747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.787500    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-kubelet-serving-ca-bundle\") pod \"ef44f77e-34bf-4c87-87d5-983882ef2035\" (UID: \"ef44f77e-34bf-4c87-87d5-983882ef2035\") "
Sep 17 00:35:56.788343 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.788223    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:56.788598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.788571    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:56.788687 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.788664    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:56.789118 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.788674    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:56.789435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.789096    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:56.789526 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.789432    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:56.789911 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.789881    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:56.790002 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.789946    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-config" (OuterVolumeSpecName: "config") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:56.790002 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.789961    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:56.790230 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.790031    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:56.790351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.790328    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:56.790442 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.790402    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:56.791252 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.791175    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-kube-api-access-zd86n" (OuterVolumeSpecName: "kube-api-access-zd86n") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "kube-api-access-zd86n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:35:56.791445 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.791417    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:56.792109 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.792087    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:56.792647 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.792614    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:56.792911 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.792882    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:35:56.792975 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.792939    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-config-out" (OuterVolumeSpecName: "config-out") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:56.792975 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.792944    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:56.799627 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799604    2576 generic.go:356] "Generic (PLEG): container finished" podID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerID="ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e" exitCode=0
Sep 17 00:35:56.799627 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799626    2576 generic.go:356] "Generic (PLEG): container finished" podID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerID="7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c" exitCode=0
Sep 17 00:35:56.799745 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799633    2576 generic.go:356] "Generic (PLEG): container finished" podID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerID="4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4" exitCode=0
Sep 17 00:35:56.799745 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799639    2576 generic.go:356] "Generic (PLEG): container finished" podID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerID="407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85" exitCode=0
Sep 17 00:35:56.799745 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799644    2576 generic.go:356] "Generic (PLEG): container finished" podID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerID="674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87" exitCode=0
Sep 17 00:35:56.799745 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799649    2576 generic.go:356] "Generic (PLEG): container finished" podID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerID="adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f" exitCode=0
Sep 17 00:35:56.799745 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerDied","Data":"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"}
Sep 17 00:35:56.799745 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799712    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:56.799969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799799    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerDied","Data":"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"}
Sep 17 00:35:56.799969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerDied","Data":"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"}
Sep 17 00:35:56.799969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799871    2576 scope.go:117] "RemoveContainer" containerID="ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"
Sep 17 00:35:56.799969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerDied","Data":"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"}
Sep 17 00:35:56.799969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799925    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerDied","Data":"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"}
Sep 17 00:35:56.799969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.799969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerDied","Data":"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"}
Sep 17 00:35:56.800215 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.800009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef44f77e-34bf-4c87-87d5-983882ef2035","Type":"ContainerDied","Data":"020c4cdfc6ae041a9dfcf9820726e3914bd5bdf6d5a5c296ba73ba1f39613168"}
Sep 17 00:35:56.803049 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.803028    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-web-config" (OuterVolumeSpecName: "web-config") pod "ef44f77e-34bf-4c87-87d5-983882ef2035" (UID: "ef44f77e-34bf-4c87-87d5-983882ef2035"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:56.807848 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.807833    2576 scope.go:117] "RemoveContainer" containerID="7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"
Sep 17 00:35:56.814343 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.814317    2576 scope.go:117] "RemoveContainer" containerID="4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"
Sep 17 00:35:56.820464 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.820447    2576 scope.go:117] "RemoveContainer" containerID="407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"
Sep 17 00:35:56.826632 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.826615    2576 scope.go:117] "RemoveContainer" containerID="674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"
Sep 17 00:35:56.833119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.833106    2576 scope.go:117] "RemoveContainer" containerID="adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"
Sep 17 00:35:56.839926 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.839911    2576 scope.go:117] "RemoveContainer" containerID="18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"
Sep 17 00:35:56.846179 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.846165    2576 scope.go:117] "RemoveContainer" containerID="ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"
Sep 17 00:35:56.846426 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:35:56.846409    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": container with ID starting with ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e not found: ID does not exist" containerID="ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"
Sep 17 00:35:56.846483 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.846431    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"} err="failed to get container status \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": rpc error: code = NotFound desc = could not find container \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": container with ID starting with ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e not found: ID does not exist"
Sep 17 00:35:56.846483 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.846451    2576 scope.go:117] "RemoveContainer" containerID="7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"
Sep 17 00:35:56.846703 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:35:56.846677    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": container with ID starting with 7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c not found: ID does not exist" containerID="7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"
Sep 17 00:35:56.846788 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.846706    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"} err="failed to get container status \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": rpc error: code = NotFound desc = could not find container \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": container with ID starting with 7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c not found: ID does not exist"
Sep 17 00:35:56.846788 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.846719    2576 scope.go:117] "RemoveContainer" containerID="4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"
Sep 17 00:35:56.846965 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:35:56.846950    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": container with ID starting with 4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4 not found: ID does not exist" containerID="4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"
Sep 17 00:35:56.847013 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.846967    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"} err="failed to get container status \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": rpc error: code = NotFound desc = could not find container \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": container with ID starting with 4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4 not found: ID does not exist"
Sep 17 00:35:56.847013 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.846979    2576 scope.go:117] "RemoveContainer" containerID="407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"
Sep 17 00:35:56.847209 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:35:56.847193    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": container with ID starting with 407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85 not found: ID does not exist" containerID="407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"
Sep 17 00:35:56.847270 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.847211    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"} err="failed to get container status \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": rpc error: code = NotFound desc = could not find container \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": container with ID starting with 407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85 not found: ID does not exist"
Sep 17 00:35:56.847270 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.847223    2576 scope.go:117] "RemoveContainer" containerID="674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"
Sep 17 00:35:56.847433 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:35:56.847421    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": container with ID starting with 674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87 not found: ID does not exist" containerID="674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"
Sep 17 00:35:56.847472 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.847435    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"} err="failed to get container status \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": rpc error: code = NotFound desc = could not find container \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": container with ID starting with 674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87 not found: ID does not exist"
Sep 17 00:35:56.847472 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.847446    2576 scope.go:117] "RemoveContainer" containerID="adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"
Sep 17 00:35:56.847636 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:35:56.847620    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": container with ID starting with adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f not found: ID does not exist" containerID="adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"
Sep 17 00:35:56.847711 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.847635    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"} err="failed to get container status \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": rpc error: code = NotFound desc = could not find container \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": container with ID starting with adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f not found: ID does not exist"
Sep 17 00:35:56.847711 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.847646    2576 scope.go:117] "RemoveContainer" containerID="18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"
Sep 17 00:35:56.847827 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:35:56.847812    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": container with ID starting with 18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9 not found: ID does not exist" containerID="18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"
Sep 17 00:35:56.847859 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.847830    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"} err="failed to get container status \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": rpc error: code = NotFound desc = could not find container \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": container with ID starting with 18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9 not found: ID does not exist"
Sep 17 00:35:56.847859 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.847840    2576 scope.go:117] "RemoveContainer" containerID="ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"
Sep 17 00:35:56.848038 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848023    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"} err="failed to get container status \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": rpc error: code = NotFound desc = could not find container \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": container with ID starting with ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e not found: ID does not exist"
Sep 17 00:35:56.848078 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848037    2576 scope.go:117] "RemoveContainer" containerID="7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"
Sep 17 00:35:56.848265 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848250    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"} err="failed to get container status \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": rpc error: code = NotFound desc = could not find container \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": container with ID starting with 7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c not found: ID does not exist"
Sep 17 00:35:56.848265 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848263    2576 scope.go:117] "RemoveContainer" containerID="4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"
Sep 17 00:35:56.848430 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848414    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"} err="failed to get container status \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": rpc error: code = NotFound desc = could not find container \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": container with ID starting with 4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4 not found: ID does not exist"
Sep 17 00:35:56.848473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848430    2576 scope.go:117] "RemoveContainer" containerID="407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"
Sep 17 00:35:56.848631 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848616    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"} err="failed to get container status \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": rpc error: code = NotFound desc = could not find container \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": container with ID starting with 407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85 not found: ID does not exist"
Sep 17 00:35:56.848703 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848631    2576 scope.go:117] "RemoveContainer" containerID="674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"
Sep 17 00:35:56.848851 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848837    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"} err="failed to get container status \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": rpc error: code = NotFound desc = could not find container \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": container with ID starting with 674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87 not found: ID does not exist"
Sep 17 00:35:56.848893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.848851    2576 scope.go:117] "RemoveContainer" containerID="adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"
Sep 17 00:35:56.849056 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849041    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"} err="failed to get container status \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": rpc error: code = NotFound desc = could not find container \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": container with ID starting with adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f not found: ID does not exist"
Sep 17 00:35:56.849056 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849054    2576 scope.go:117] "RemoveContainer" containerID="18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"
Sep 17 00:35:56.849293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849274    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"} err="failed to get container status \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": rpc error: code = NotFound desc = could not find container \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": container with ID starting with 18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9 not found: ID does not exist"
Sep 17 00:35:56.849341 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849294    2576 scope.go:117] "RemoveContainer" containerID="ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"
Sep 17 00:35:56.849494 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849478    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"} err="failed to get container status \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": rpc error: code = NotFound desc = could not find container \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": container with ID starting with ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e not found: ID does not exist"
Sep 17 00:35:56.849494 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849493    2576 scope.go:117] "RemoveContainer" containerID="7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"
Sep 17 00:35:56.849711 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849696    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"} err="failed to get container status \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": rpc error: code = NotFound desc = could not find container \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": container with ID starting with 7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c not found: ID does not exist"
Sep 17 00:35:56.849754 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849710    2576 scope.go:117] "RemoveContainer" containerID="4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"
Sep 17 00:35:56.849900 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849885    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"} err="failed to get container status \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": rpc error: code = NotFound desc = could not find container \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": container with ID starting with 4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4 not found: ID does not exist"
Sep 17 00:35:56.849900 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.849899    2576 scope.go:117] "RemoveContainer" containerID="407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"
Sep 17 00:35:56.850094 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850079    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"} err="failed to get container status \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": rpc error: code = NotFound desc = could not find container \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": container with ID starting with 407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85 not found: ID does not exist"
Sep 17 00:35:56.850161 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850094    2576 scope.go:117] "RemoveContainer" containerID="674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"
Sep 17 00:35:56.850318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850303    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"} err="failed to get container status \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": rpc error: code = NotFound desc = could not find container \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": container with ID starting with 674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87 not found: ID does not exist"
Sep 17 00:35:56.850318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850316    2576 scope.go:117] "RemoveContainer" containerID="adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"
Sep 17 00:35:56.850503 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850489    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"} err="failed to get container status \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": rpc error: code = NotFound desc = could not find container \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": container with ID starting with adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f not found: ID does not exist"
Sep 17 00:35:56.850503 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850502    2576 scope.go:117] "RemoveContainer" containerID="18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"
Sep 17 00:35:56.850752 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850734    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"} err="failed to get container status \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": rpc error: code = NotFound desc = could not find container \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": container with ID starting with 18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9 not found: ID does not exist"
Sep 17 00:35:56.850825 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850753    2576 scope.go:117] "RemoveContainer" containerID="ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"
Sep 17 00:35:56.850991 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850972    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"} err="failed to get container status \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": rpc error: code = NotFound desc = could not find container \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": container with ID starting with ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e not found: ID does not exist"
Sep 17 00:35:56.851059 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.850994    2576 scope.go:117] "RemoveContainer" containerID="7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"
Sep 17 00:35:56.851279 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.851262    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"} err="failed to get container status \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": rpc error: code = NotFound desc = could not find container \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": container with ID starting with 7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c not found: ID does not exist"
Sep 17 00:35:56.851330 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.851281    2576 scope.go:117] "RemoveContainer" containerID="4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"
Sep 17 00:35:56.851473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.851455    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"} err="failed to get container status \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": rpc error: code = NotFound desc = could not find container \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": container with ID starting with 4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4 not found: ID does not exist"
Sep 17 00:35:56.851536 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.851474    2576 scope.go:117] "RemoveContainer" containerID="407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"
Sep 17 00:35:56.851718 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.851702    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"} err="failed to get container status \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": rpc error: code = NotFound desc = could not find container \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": container with ID starting with 407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85 not found: ID does not exist"
Sep 17 00:35:56.851767 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.851717    2576 scope.go:117] "RemoveContainer" containerID="674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"
Sep 17 00:35:56.851946 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.851927    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"} err="failed to get container status \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": rpc error: code = NotFound desc = could not find container \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": container with ID starting with 674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87 not found: ID does not exist"
Sep 17 00:35:56.851987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.851949    2576 scope.go:117] "RemoveContainer" containerID="adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"
Sep 17 00:35:56.852165 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.852133    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"} err="failed to get container status \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": rpc error: code = NotFound desc = could not find container \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": container with ID starting with adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f not found: ID does not exist"
Sep 17 00:35:56.852226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.852166    2576 scope.go:117] "RemoveContainer" containerID="18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"
Sep 17 00:35:56.852378 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.852360    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"} err="failed to get container status \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": rpc error: code = NotFound desc = could not find container \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": container with ID starting with 18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9 not found: ID does not exist"
Sep 17 00:35:56.852415 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.852378    2576 scope.go:117] "RemoveContainer" containerID="ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"
Sep 17 00:35:56.852597 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.852580    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"} err="failed to get container status \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": rpc error: code = NotFound desc = could not find container \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": container with ID starting with ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e not found: ID does not exist"
Sep 17 00:35:56.852639 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.852598    2576 scope.go:117] "RemoveContainer" containerID="7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"
Sep 17 00:35:56.852818 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.852802    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"} err="failed to get container status \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": rpc error: code = NotFound desc = could not find container \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": container with ID starting with 7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c not found: ID does not exist"
Sep 17 00:35:56.852818 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.852818    2576 scope.go:117] "RemoveContainer" containerID="4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"
Sep 17 00:35:56.853031 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853013    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"} err="failed to get container status \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": rpc error: code = NotFound desc = could not find container \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": container with ID starting with 4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4 not found: ID does not exist"
Sep 17 00:35:56.853073 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853032    2576 scope.go:117] "RemoveContainer" containerID="407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"
Sep 17 00:35:56.853254 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853238    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"} err="failed to get container status \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": rpc error: code = NotFound desc = could not find container \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": container with ID starting with 407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85 not found: ID does not exist"
Sep 17 00:35:56.853324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853254    2576 scope.go:117] "RemoveContainer" containerID="674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"
Sep 17 00:35:56.853452 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853430    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"} err="failed to get container status \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": rpc error: code = NotFound desc = could not find container \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": container with ID starting with 674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87 not found: ID does not exist"
Sep 17 00:35:56.853489 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853453    2576 scope.go:117] "RemoveContainer" containerID="adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"
Sep 17 00:35:56.853694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853663    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"} err="failed to get container status \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": rpc error: code = NotFound desc = could not find container \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": container with ID starting with adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f not found: ID does not exist"
Sep 17 00:35:56.853694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853691    2576 scope.go:117] "RemoveContainer" containerID="18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"
Sep 17 00:35:56.853905 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853889    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"} err="failed to get container status \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": rpc error: code = NotFound desc = could not find container \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": container with ID starting with 18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9 not found: ID does not exist"
Sep 17 00:35:56.853954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.853906    2576 scope.go:117] "RemoveContainer" containerID="ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"
Sep 17 00:35:56.854115 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854099    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e"} err="failed to get container status \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": rpc error: code = NotFound desc = could not find container \"ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e\": container with ID starting with ffabe37a2d041ef6c7580939b425a812e1c725957eb8a76ecf9f0a3de80ad16e not found: ID does not exist"
Sep 17 00:35:56.854181 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854115    2576 scope.go:117] "RemoveContainer" containerID="7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"
Sep 17 00:35:56.854334 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854318    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c"} err="failed to get container status \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": rpc error: code = NotFound desc = could not find container \"7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c\": container with ID starting with 7adc43884d0a62f301f8496ccc14fb34c3c2012198b002dfabb90fc237de309c not found: ID does not exist"
Sep 17 00:35:56.854375 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854334    2576 scope.go:117] "RemoveContainer" containerID="4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"
Sep 17 00:35:56.854525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854509    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4"} err="failed to get container status \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": rpc error: code = NotFound desc = could not find container \"4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4\": container with ID starting with 4dbdea8099220ba17c6c9548c17a1cb570dd53de30724ed0fa45e8a98af77ec4 not found: ID does not exist"
Sep 17 00:35:56.854577 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854525    2576 scope.go:117] "RemoveContainer" containerID="407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"
Sep 17 00:35:56.854760 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854742    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85"} err="failed to get container status \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": rpc error: code = NotFound desc = could not find container \"407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85\": container with ID starting with 407f5c78d724a2908a69d56b71326a60f907b17fe6f64c4122da5efd79414a85 not found: ID does not exist"
Sep 17 00:35:56.854837 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854761    2576 scope.go:117] "RemoveContainer" containerID="674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"
Sep 17 00:35:56.854991 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854973    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87"} err="failed to get container status \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": rpc error: code = NotFound desc = could not find container \"674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87\": container with ID starting with 674f513847fc4a88a02c824edbb08f6f13c6154e130b276e750207c61ffa0f87 not found: ID does not exist"
Sep 17 00:35:56.855037 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.854991    2576 scope.go:117] "RemoveContainer" containerID="adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"
Sep 17 00:35:56.855220 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.855204    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f"} err="failed to get container status \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": rpc error: code = NotFound desc = could not find container \"adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f\": container with ID starting with adf3d0f57e4bf200cdd2b19257dd79008a32c8741684a2e697c2568b8a6b644f not found: ID does not exist"
Sep 17 00:35:56.855263 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.855220    2576 scope.go:117] "RemoveContainer" containerID="18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"
Sep 17 00:35:56.855443 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.855426    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9"} err="failed to get container status \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": rpc error: code = NotFound desc = could not find container \"18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9\": container with ID starting with 18c681bdeeb25ac96f73c53f2473ff1c93eeaf3e3df1040a4a41442eaf0186d9 not found: ID does not exist"
Sep 17 00:35:56.888372 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888351    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888372 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888369    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888379    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-web-config\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888388    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-config-out\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888396    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888405    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888414    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888422    2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-grpc-tls\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888430    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888438    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888446    2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-config\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888453    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888462    2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888470    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-k8s-db\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888478    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd86n\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-kube-api-access-zd86n\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888485    2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-kube-rbac-proxy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888494    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef44f77e-34bf-4c87-87d5-983882ef2035-tls-assets\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888502    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-configmap-metrics-client-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888510    2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef44f77e-34bf-4c87-87d5-983882ef2035-secret-metrics-client-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:56.888886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:56.888518    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef44f77e-34bf-4c87-87d5-983882ef2035-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:35:57.124292 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.124265    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 00:35:57.130895 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.130870    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 00:35:57.152733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152709    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 00:35:57.152939 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152928    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="config-reloader"
Sep 17 00:35:57.152985 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152940    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="config-reloader"
Sep 17 00:35:57.152985 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152953    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy-thanos"
Sep 17 00:35:57.152985 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152958    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy-thanos"
Sep 17 00:35:57.152985 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152964    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="init-config-reloader"
Sep 17 00:35:57.152985 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152970    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="init-config-reloader"
Sep 17 00:35:57.152985 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152975    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy-web"
Sep 17 00:35:57.152985 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152980    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy-web"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152989    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="434bf0d4-94c9-4c13-bc41-7f892a841e8e" containerName="console"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.152994    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="434bf0d4-94c9-4c13-bc41-7f892a841e8e" containerName="console"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153001    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153006    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153013    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="prometheus"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153018    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="prometheus"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153024    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="thanos-sidecar"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153029    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="thanos-sidecar"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153035    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="853a6578-a7f7-4ed0-9331-e9877dadf9ce" containerName="registry"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153041    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="853a6578-a7f7-4ed0-9331-e9877dadf9ce" containerName="registry"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153079    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="434bf0d4-94c9-4c13-bc41-7f892a841e8e" containerName="console"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153087    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="thanos-sidecar"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153092    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy-thanos"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153097    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy-web"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153102    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="853a6578-a7f7-4ed0-9331-e9877dadf9ce" containerName="registry"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153107    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="config-reloader"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153111    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="kube-rbac-proxy"
Sep 17 00:35:57.153280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.153116    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" containerName="prometheus"
Sep 17 00:35:57.158022 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.158006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.160559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.160540    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 00:35:57.160876 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.160854    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 00:35:57.160976 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.160857    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 00:35:57.160976 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.160970    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 00:35:57.161126 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.161106    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 00:35:57.161281 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.161261    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-74pvk\""
Sep 17 00:35:57.161344 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.161279    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 00:35:57.161344 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.161265    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 00:35:57.161459 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.161441    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-46lflvoio0kkg\""
Sep 17 00:35:57.161548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.161533    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 00:35:57.162202 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.162172    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 00:35:57.162202 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.162173    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 00:35:57.162369 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.162235    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 00:35:57.162369 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.162333    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 00:35:57.162858 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.162837    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 00:35:57.164216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.164199    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 00:35:57.165957 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.165937    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 00:35:57.170465 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.170446    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 00:35:57.291753 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291732    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.291844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e787bd95-8de8-4109-b922-f0730905c907-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.291844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.291844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.291844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.291844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-config\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.291993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.292017    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.292038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-web-config\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.292076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.292091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.292107    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9tc6\" (UniqueName: \"kubernetes.io/projected/e787bd95-8de8-4109-b922-f0730905c907-kube-api-access-x9tc6\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.292137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e787bd95-8de8-4109-b922-f0730905c907-config-out\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.292226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.292170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.393231 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.393165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.393231 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.393214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e787bd95-8de8-4109-b922-f0730905c907-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.393360 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.393320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.393360 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.393343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.393446 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.393363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.393446 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.393386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.393604 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.393522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.393604 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.393577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.393705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.393619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-config\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.394310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.394081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.394310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.394121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.394310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.394159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.394310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.394173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.394310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.394193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.394731    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.394778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.394820    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.394903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.395004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.395041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-web-config\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.395084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.395109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.395134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tc6\" (UniqueName: \"kubernetes.io/projected/e787bd95-8de8-4109-b922-f0730905c907-kube-api-access-x9tc6\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.395191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e787bd95-8de8-4109-b922-f0730905c907-config-out\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.395476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.395220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.396943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.396074    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.396943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.396597    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e787bd95-8de8-4109-b922-f0730905c907-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.396943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.396665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.396943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.396726    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.396943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.396805    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.396943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.396873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.397400 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.397379    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-config\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.397538 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.397513    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.398432 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.398406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.398995 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.398973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.399463 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.399435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e787bd95-8de8-4109-b922-f0730905c907-config-out\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.399612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.399590    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.399733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.399712    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e787bd95-8de8-4109-b922-f0730905c907-web-config\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.400126 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.400108    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e787bd95-8de8-4109-b922-f0730905c907-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.404480 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.404464    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tc6\" (UniqueName: \"kubernetes.io/projected/e787bd95-8de8-4109-b922-f0730905c907-kube-api-access-x9tc6\") pod \"prometheus-k8s-0\" (UID: \"e787bd95-8de8-4109-b922-f0730905c907\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.468038 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.468018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:35:57.746082 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.746007    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 00:35:57.748257 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:35:57.748231    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode787bd95_8de8_4109_b922_f0730905c907.slice/crio-f22eba9d580eec2938b14a238f81f20c876b5a4f7c349604ca124b689298963d WatchSource:0}: Error finding container f22eba9d580eec2938b14a238f81f20c876b5a4f7c349604ca124b689298963d: Status 404 returned error can't find the container with id f22eba9d580eec2938b14a238f81f20c876b5a4f7c349604ca124b689298963d
Sep 17 00:35:57.807048 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:57.807024    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e787bd95-8de8-4109-b922-f0730905c907","Type":"ContainerStarted","Data":"f22eba9d580eec2938b14a238f81f20c876b5a4f7c349604ca124b689298963d"}
Sep 17 00:35:58.587262 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:58.587228    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef44f77e-34bf-4c87-87d5-983882ef2035" path="/var/lib/kubelet/pods/ef44f77e-34bf-4c87-87d5-983882ef2035/volumes"
Sep 17 00:35:58.812058 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:58.812022    2576 generic.go:356] "Generic (PLEG): container finished" podID="e787bd95-8de8-4109-b922-f0730905c907" containerID="db206b1c6863b5ba06b1f4166b71996a018081cc6a241d2af8ac647d56d1a2d7" exitCode=0
Sep 17 00:35:58.812207 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:58.812108    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e787bd95-8de8-4109-b922-f0730905c907","Type":"ContainerDied","Data":"db206b1c6863b5ba06b1f4166b71996a018081cc6a241d2af8ac647d56d1a2d7"}
Sep 17 00:35:59.306035 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:59.306001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:59.357473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:59.357387    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:35:59.820079 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:59.820046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e787bd95-8de8-4109-b922-f0730905c907","Type":"ContainerStarted","Data":"b2e3552e10b77ba79d5530f04372fc76962c3b1a4dee19bf27256028bec424db"}
Sep 17 00:35:59.820439 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:59.820133    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e787bd95-8de8-4109-b922-f0730905c907","Type":"ContainerStarted","Data":"ee8b68cc1127e5b7487bef71080212dc58b65a5b9240513768906c9a375ed4b5"}
Sep 17 00:35:59.820439 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:59.820176    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e787bd95-8de8-4109-b922-f0730905c907","Type":"ContainerStarted","Data":"497c1010b04b2da71990ebd67a45c1bd813f8d201a6906973aee89ed3f47aad6"}
Sep 17 00:35:59.820439 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:59.820201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e787bd95-8de8-4109-b922-f0730905c907","Type":"ContainerStarted","Data":"01a35d25153be693aa2f9e62733fc37a08111966bdb9ba2b0714fadbdc76b518"}
Sep 17 00:35:59.820439 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:59.820226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e787bd95-8de8-4109-b922-f0730905c907","Type":"ContainerStarted","Data":"ea778352115f493eaf18998c3c4835effe94b54df6fd535bedc7a678908b8475"}
Sep 17 00:35:59.820439 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:59.820274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e787bd95-8de8-4109-b922-f0730905c907","Type":"ContainerStarted","Data":"6cf9162a2b7eed373e504ea1447a645b59100cf9ff8898dda45fc6f3e68a31af"}
Sep 17 00:35:59.848658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:35:59.848609    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.826501619 podStartE2EDuration="2.848594325s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.092706ms" isStatefulPod=false podCreationTimestamp="2026-09-17 00:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:35:59.846812647 +0000 UTC m=+355.802409011" watchObservedRunningTime="2026-09-17 00:35:59.848594325 +0000 UTC m=+355.804190666"
Sep 17 00:36:02.468349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:02.468308    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:36:03.288887 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:03.288840    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:36:03.338013 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:03.337982    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:36:10.550899 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:10.550869    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:36:10.602032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:10.602002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:36:10.657784 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:10.657763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:36:10.703752 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:10.703724    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:36:33.770452 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:33.770415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:36:33.819077 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:33.819051    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:36:57.468736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:57.468694    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:36:57.486098 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:57.486072    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:36:58.057394 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:36:58.057369    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 00:37:06.227043 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:06.227007    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:06.294504 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:06.294474    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:12.583323 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:12.583277    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:12.635696 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:12.635661    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:12.730352 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:12.730320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:13.331202 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:13.331162    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:13.381820 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:13.381783    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:18.206103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:18.206068    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:18.257030 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:18.256981    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:23.760081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:23.760042    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:23.816512 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:23.816484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:23.890873 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:23.890839    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:23.991712 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:23.991678    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:34.602521 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:34.602424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:37:34.701103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:37:34.701060    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:38:18.677610 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:38:18.677571    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:38:42.017503 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:38:42.017471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:38:42.119538 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:38:42.119503    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:02.349937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:02.349855    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:02.404755 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:02.404719    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:08.610464 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:08.610427    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:09.389009 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:09.388970    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:12.218845 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:12.218809    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:12.269674 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:12.269638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:15.444926 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:15.444875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:15.552888 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:15.552862    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:17.433351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:17.433319    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:17.530774 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:17.530747    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:17.845041 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:17.845013    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:18.232567 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:18.232492    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:18.611245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:18.611214    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:18.661946 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:18.661917    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:24.769473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:24.769439    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:26.122708 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:26.122675    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:36.426490 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:36.426452    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:36.525358 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:36.525330    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:38.144818 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.144785    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"]
Sep 17 00:39:38.147864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.147849    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"
Sep 17 00:39:38.150375 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.150352    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tj9l7\""
Sep 17 00:39:38.150375 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.150373    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 00:39:38.151275 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.151261    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 00:39:38.154766 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.154614    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"]
Sep 17 00:39:38.251560 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.251520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebbc5c7a-af9b-4d4d-b17f-7cae1718d303-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qjw2x\" (UID: \"ebbc5c7a-af9b-4d4d-b17f-7cae1718d303\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"
Sep 17 00:39:38.251739 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.251564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7pf8\" (UniqueName: \"kubernetes.io/projected/ebbc5c7a-af9b-4d4d-b17f-7cae1718d303-kube-api-access-k7pf8\") pod \"openshift-lws-operator-fd8ccff4c-qjw2x\" (UID: \"ebbc5c7a-af9b-4d4d-b17f-7cae1718d303\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"
Sep 17 00:39:38.352081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.352050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebbc5c7a-af9b-4d4d-b17f-7cae1718d303-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qjw2x\" (UID: \"ebbc5c7a-af9b-4d4d-b17f-7cae1718d303\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"
Sep 17 00:39:38.352237 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.352095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7pf8\" (UniqueName: \"kubernetes.io/projected/ebbc5c7a-af9b-4d4d-b17f-7cae1718d303-kube-api-access-k7pf8\") pod \"openshift-lws-operator-fd8ccff4c-qjw2x\" (UID: \"ebbc5c7a-af9b-4d4d-b17f-7cae1718d303\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"
Sep 17 00:39:38.352454 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.352436    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebbc5c7a-af9b-4d4d-b17f-7cae1718d303-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qjw2x\" (UID: \"ebbc5c7a-af9b-4d4d-b17f-7cae1718d303\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"
Sep 17 00:39:38.360623 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.360591    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7pf8\" (UniqueName: \"kubernetes.io/projected/ebbc5c7a-af9b-4d4d-b17f-7cae1718d303-kube-api-access-k7pf8\") pod \"openshift-lws-operator-fd8ccff4c-qjw2x\" (UID: \"ebbc5c7a-af9b-4d4d-b17f-7cae1718d303\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"
Sep 17 00:39:38.456937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.456719    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"
Sep 17 00:39:38.729681 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.729596    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x"]
Sep 17 00:39:38.732628 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:39:38.732593    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebbc5c7a_af9b_4d4d_b17f_7cae1718d303.slice/crio-71a0f63733b1ebc08b7c8e88b13772cec77e8c2953edae819a875a2c5ace549a WatchSource:0}: Error finding container 71a0f63733b1ebc08b7c8e88b13772cec77e8c2953edae819a875a2c5ace549a: Status 404 returned error can't find the container with id 71a0f63733b1ebc08b7c8e88b13772cec77e8c2953edae819a875a2c5ace549a
Sep 17 00:39:38.734040 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:38.734023    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:39:39.621344 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:39.621299    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x" event={"ID":"ebbc5c7a-af9b-4d4d-b17f-7cae1718d303","Type":"ContainerStarted","Data":"71a0f63733b1ebc08b7c8e88b13772cec77e8c2953edae819a875a2c5ace549a"}
Sep 17 00:39:40.988517 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:40.988464    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:41.042098 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:41.042058    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:41.136138 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:41.136112    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:41.236605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:41.236578    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:41.631533 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:41.631496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x" event={"ID":"ebbc5c7a-af9b-4d4d-b17f-7cae1718d303","Type":"ContainerStarted","Data":"7f75e4b06633bbdacb24bdbfd9646da52979522c1e369822b399ae76af48aef2"}
Sep 17 00:39:41.648099 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:41.648047    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qjw2x" podStartSLOduration=1.256175393 podStartE2EDuration="3.648030946s" totalImagesPullingTime="2.391855553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:39:41.647436329 +0000 UTC m=+577.603032672" watchObservedRunningTime="2026-09-17 00:39:41.648030946 +0000 UTC m=+577.603627287"
Sep 17 00:39:41.727054 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:41.727021    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:41.828291 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:41.828251    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:42.318854 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:42.318823    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:42.419909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:42.419828    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:44.140074 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:44.140047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:52.322323 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:52.322286    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:52.422669 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:52.422642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:52.469155 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:52.469101    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:54.378373 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.378341    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:54.428649 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.428619    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"]
Sep 17 00:39:54.432003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.431983    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.434581 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.434560    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 00:39:54.434678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.434567    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6cghx\""
Sep 17 00:39:54.435989 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.435967    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 00:39:54.436107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.436034    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 00:39:54.441569 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.441546    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"]
Sep 17 00:39:54.457203 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.457178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:39:54.582720 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.582688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-manager-config\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.582901 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.582743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-metrics-cert\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.582901 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.582811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-cert\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.583002 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.582911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc7ws\" (UniqueName: \"kubernetes.io/projected/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-kube-api-access-kc7ws\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.683673 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.683582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc7ws\" (UniqueName: \"kubernetes.io/projected/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-kube-api-access-kc7ws\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.683673 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.683625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-manager-config\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.683909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.683676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-metrics-cert\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.683909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.683733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-cert\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.684451 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.684423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-manager-config\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.686531 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.686498    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-metrics-cert\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.686947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.686926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-cert\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.697157 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.697124    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc7ws\" (UniqueName: \"kubernetes.io/projected/f8be64ce-0dcc-4d26-9113-5efa0ef62a7b-kube-api-access-kc7ws\") pod \"lws-controller-manager-68945d9444-5cvgh\" (UID: \"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:54.741620 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:54.741596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:55.013830 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:55.013773    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"]
Sep 17 00:39:55.016344 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:39:55.016316    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8be64ce_0dcc_4d26_9113_5efa0ef62a7b.slice/crio-bb17b570367a6f740a8021e0bedf7d8c4a66ca67ce99af1649429da585ec4ad7 WatchSource:0}: Error finding container bb17b570367a6f740a8021e0bedf7d8c4a66ca67ce99af1649429da585ec4ad7: Status 404 returned error can't find the container with id bb17b570367a6f740a8021e0bedf7d8c4a66ca67ce99af1649429da585ec4ad7
Sep 17 00:39:55.692156 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:55.692107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh" event={"ID":"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b","Type":"ContainerStarted","Data":"bb17b570367a6f740a8021e0bedf7d8c4a66ca67ce99af1649429da585ec4ad7"}
Sep 17 00:39:57.702783 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:57.702746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh" event={"ID":"f8be64ce-0dcc-4d26-9113-5efa0ef62a7b","Type":"ContainerStarted","Data":"e00beac5077712bdda92fd25155ef225b9d265646c28294834ac658321452cc3"}
Sep 17 00:39:57.703201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:57.702880    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:39:57.719190 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:39:57.719120    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh" podStartSLOduration=1.8499638919999999 podStartE2EDuration="3.719103522s" totalImagesPullingTime="1.86913963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:39:57.717812224 +0000 UTC m=+593.673408566" watchObservedRunningTime="2026-09-17 00:39:57.719103522 +0000 UTC m=+593.674699919"
Sep 17 00:40:00.025044 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:00.025012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:00.122289 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:00.122252    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:00.832708 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:00.832675    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:01.080664 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.080635    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:01.178927 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.178849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:01.295519 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.295485    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"]
Sep 17 00:40:01.298822 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.298808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.302961 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.302942    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 00:40:01.303076 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.302990    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 00:40:01.303345 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.303331    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 00:40:01.303406 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.303335    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 00:40:01.303764 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.303751    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mmxsx\""
Sep 17 00:40:01.326467 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.326441    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"]
Sep 17 00:40:01.337200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.337177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnfc\" (UniqueName: \"kubernetes.io/projected/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-kube-api-access-pfnfc\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.337340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.337223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.337340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.337242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-tmp\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.337340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.337263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-manifests\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.337340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.337326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.428455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.428418    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:01.438595 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.438540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.438595 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.438570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-tmp\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.438595 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.438591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-manifests\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.438748 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.438710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.438820 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.438802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnfc\" (UniqueName: \"kubernetes.io/projected/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-kube-api-access-pfnfc\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.439030 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.439008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-manifests\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.440967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.440946    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-tmp\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.441284 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.441266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.441326 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.441306    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.445986 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.445965    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnfc\" (UniqueName: \"kubernetes.io/projected/b3b2214f-12a7-4d14-a367-0f6410d9a4fc-kube-api-access-pfnfc\") pod \"opendatahub-operator-controller-manager-596c59f645-k4b9n\" (UID: \"b3b2214f-12a7-4d14-a367-0f6410d9a4fc\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.528931 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.528896    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:01.608901 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.608866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:01.628895 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.628867    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:01.686116 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.686079    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:01.787785 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.787758    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:01.854386 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.854355    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:01.884558 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:01.884535    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"]
Sep 17 00:40:01.886411 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:40:01.886385    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b2214f_12a7_4d14_a367_0f6410d9a4fc.slice/crio-54ab05430dd5ed3dc21fcaec00d2b33e1565f1a4409f2c3ab90870d0dceced2f WatchSource:0}: Error finding container 54ab05430dd5ed3dc21fcaec00d2b33e1565f1a4409f2c3ab90870d0dceced2f: Status 404 returned error can't find the container with id 54ab05430dd5ed3dc21fcaec00d2b33e1565f1a4409f2c3ab90870d0dceced2f
Sep 17 00:40:02.726200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:02.726138    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n" event={"ID":"b3b2214f-12a7-4d14-a367-0f6410d9a4fc","Type":"ContainerStarted","Data":"54ab05430dd5ed3dc21fcaec00d2b33e1565f1a4409f2c3ab90870d0dceced2f"}
Sep 17 00:40:04.935843 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:04.935818    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:40:04.938250 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:04.935819    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:40:05.742364 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:05.742326    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n" event={"ID":"b3b2214f-12a7-4d14-a367-0f6410d9a4fc","Type":"ContainerStarted","Data":"a0a9629ad261bfbc69a920c8b6a088f12d9a25db424629d768704da96df8bfdf"}
Sep 17 00:40:06.748018 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:06.747978    2576 generic.go:356] "Generic (PLEG): container finished" podID="b3b2214f-12a7-4d14-a367-0f6410d9a4fc" containerID="a0a9629ad261bfbc69a920c8b6a088f12d9a25db424629d768704da96df8bfdf" exitCode=0
Sep 17 00:40:06.748466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:06.748070    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n" event={"ID":"b3b2214f-12a7-4d14-a367-0f6410d9a4fc","Type":"ContainerDied","Data":"a0a9629ad261bfbc69a920c8b6a088f12d9a25db424629d768704da96df8bfdf"}
Sep 17 00:40:07.754043 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:07.754008    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n" event={"ID":"b3b2214f-12a7-4d14-a367-0f6410d9a4fc","Type":"ContainerStarted","Data":"e8f1834bc8c1e835632efc04e59e946bd44c5a05ce6c6814abc56ce93642e5ec"}
Sep 17 00:40:07.754493 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:07.754276    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:07.783852 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:07.783800    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n" podStartSLOduration=2.241813876 podStartE2EDuration="6.783785089s" totalImagesPullingTime="3.525112898s" totalInitContainerRuntime="1.016858315s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:40:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:40:07.782640878 +0000 UTC m=+603.738237218" watchObservedRunningTime="2026-09-17 00:40:07.783785089 +0000 UTC m=+603.739381429"
Sep 17 00:40:08.709564 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:08.709534    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68945d9444-5cvgh"
Sep 17 00:40:13.427768 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:13.427731    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:13.530909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:13.530872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:18.761362 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:18.761332    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-k4b9n"
Sep 17 00:40:20.193158 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:20.193111    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:20.304863 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:20.304832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:20.515302 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:20.515220    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:20.819527 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:20.819491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:21.797572 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:21.797540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:21.856349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:21.856317    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:22.728752 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:22.728716    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:27.004622 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.004543    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:27.104936 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.104906    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:27.265190 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.265079    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"]
Sep 17 00:40:27.274469 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.274447    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.276940 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.276916    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qb7rb\""
Sep 17 00:40:27.277058 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.276960    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 00:40:27.280367 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.280342    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"]
Sep 17 00:40:27.297103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.297074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:27.357122 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.357092    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.357122 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.357126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.357374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.357169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.357374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.357186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwt7q\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-kube-api-access-lwt7q\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.357374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.357261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.357374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.357297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.357374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.357314    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.357374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.357357    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.357579 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.357381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.382755 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.382730    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:27.444639 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.444601    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:27.458461 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.458440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.458553 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.458466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwt7q\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-kube-api-access-lwt7q\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.458553 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.458499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.458553 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.458520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.458553 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.458537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.458764 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.458563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.458764 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.458596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.458764 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.458656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.458916 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.458791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.459046 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.459022    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.459163 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.459066    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.459163 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.459090    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.459163 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.459129    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.459327 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.459303    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.461132 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.461102    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.461332 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.461314    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.465792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.465775    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.465856 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.465822    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwt7q\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-kube-api-access-lwt7q\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.587602 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.587570    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:27.861392 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:27.861367    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"]
Sep 17 00:40:27.863794 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:40:27.863760    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730a7423_3f23_4b9a_add0_9c5e3aa6d41e.slice/crio-6d9a9ab6265010c4b69405716558f4c67064d6c71e05b0361b0ac09145ce281e WatchSource:0}: Error finding container 6d9a9ab6265010c4b69405716558f4c67064d6c71e05b0361b0ac09145ce281e: Status 404 returned error can't find the container with id 6d9a9ab6265010c4b69405716558f4c67064d6c71e05b0361b0ac09145ce281e
Sep 17 00:40:28.844587 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:28.844557    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j" event={"ID":"730a7423-3f23-4b9a-add0-9c5e3aa6d41e","Type":"ContainerStarted","Data":"6d9a9ab6265010c4b69405716558f4c67064d6c71e05b0361b0ac09145ce281e"}
Sep 17 00:40:31.177896 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.177855    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h5kxq"]
Sep 17 00:40:31.182033 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.182008    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:31.184478 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.184457    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 17 00:40:31.184603 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.184520    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kwd7k\""
Sep 17 00:40:31.192503 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.192481    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h5kxq"]
Sep 17 00:40:31.253598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.253567    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:31.295270 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.295238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/009e1fc3-a69e-47c3-924a-ab5adc7066f5-cert\") pod \"odh-model-controller-7998c67dc4-h5kxq\" (UID: \"009e1fc3-a69e-47c3-924a-ab5adc7066f5\") " pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:31.295435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.295301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxnw\" (UniqueName: \"kubernetes.io/projected/009e1fc3-a69e-47c3-924a-ab5adc7066f5-kube-api-access-smxnw\") pod \"odh-model-controller-7998c67dc4-h5kxq\" (UID: \"009e1fc3-a69e-47c3-924a-ab5adc7066f5\") " pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:31.359708 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.359678    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:31.396217 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.396190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxnw\" (UniqueName: \"kubernetes.io/projected/009e1fc3-a69e-47c3-924a-ab5adc7066f5-kube-api-access-smxnw\") pod \"odh-model-controller-7998c67dc4-h5kxq\" (UID: \"009e1fc3-a69e-47c3-924a-ab5adc7066f5\") " pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:31.396375 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.396275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/009e1fc3-a69e-47c3-924a-ab5adc7066f5-cert\") pod \"odh-model-controller-7998c67dc4-h5kxq\" (UID: \"009e1fc3-a69e-47c3-924a-ab5adc7066f5\") " pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:31.396441 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:40:31.396376    2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 00:40:31.396497 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:40:31.396442    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/009e1fc3-a69e-47c3-924a-ab5adc7066f5-cert podName:009e1fc3-a69e-47c3-924a-ab5adc7066f5 nodeName:}" failed. No retries permitted until 2026-09-17 00:40:31.896423758 +0000 UTC m=+627.852020077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/009e1fc3-a69e-47c3-924a-ab5adc7066f5-cert") pod "odh-model-controller-7998c67dc4-h5kxq" (UID: "009e1fc3-a69e-47c3-924a-ab5adc7066f5") : secret "odh-model-controller-webhook-cert" not found
Sep 17 00:40:31.417321 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.417292    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxnw\" (UniqueName: \"kubernetes.io/projected/009e1fc3-a69e-47c3-924a-ab5adc7066f5-kube-api-access-smxnw\") pod \"odh-model-controller-7998c67dc4-h5kxq\" (UID: \"009e1fc3-a69e-47c3-924a-ab5adc7066f5\") " pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:31.466214 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.466117    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:31.554739 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.554707    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:31.840925 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.840896    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hk8l5"]
Sep 17 00:40:31.845367 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.845349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:31.847729 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.847705    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 17 00:40:31.852559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.852536    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hk8l5"]
Sep 17 00:40:31.858126 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.858095    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:31.900501 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.900467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/009e1fc3-a69e-47c3-924a-ab5adc7066f5-cert\") pod \"odh-model-controller-7998c67dc4-h5kxq\" (UID: \"009e1fc3-a69e-47c3-924a-ab5adc7066f5\") " pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:31.903016 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:31.902987    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/009e1fc3-a69e-47c3-924a-ab5adc7066f5-cert\") pod \"odh-model-controller-7998c67dc4-h5kxq\" (UID: \"009e1fc3-a69e-47c3-924a-ab5adc7066f5\") " pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:32.001103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.001058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35441bf9-fae9-40d9-90a4-443a82ef0921-tls-certs\") pod \"model-serving-api-54d89bd5fc-hk8l5\" (UID: \"35441bf9-fae9-40d9-90a4-443a82ef0921\") " pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:32.001296 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.001117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64fw5\" (UniqueName: \"kubernetes.io/projected/35441bf9-fae9-40d9-90a4-443a82ef0921-kube-api-access-64fw5\") pod \"model-serving-api-54d89bd5fc-hk8l5\" (UID: \"35441bf9-fae9-40d9-90a4-443a82ef0921\") " pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:32.093361 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.093270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:32.101649 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.101625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35441bf9-fae9-40d9-90a4-443a82ef0921-tls-certs\") pod \"model-serving-api-54d89bd5fc-hk8l5\" (UID: \"35441bf9-fae9-40d9-90a4-443a82ef0921\") " pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:32.101772 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.101660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64fw5\" (UniqueName: \"kubernetes.io/projected/35441bf9-fae9-40d9-90a4-443a82ef0921-kube-api-access-64fw5\") pod \"model-serving-api-54d89bd5fc-hk8l5\" (UID: \"35441bf9-fae9-40d9-90a4-443a82ef0921\") " pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:32.101846 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:40:32.101796    2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 00:40:32.101903 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:40:32.101881    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35441bf9-fae9-40d9-90a4-443a82ef0921-tls-certs podName:35441bf9-fae9-40d9-90a4-443a82ef0921 nodeName:}" failed. No retries permitted until 2026-09-17 00:40:32.601859191 +0000 UTC m=+628.557455516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/35441bf9-fae9-40d9-90a4-443a82ef0921-tls-certs") pod "model-serving-api-54d89bd5fc-hk8l5" (UID: "35441bf9-fae9-40d9-90a4-443a82ef0921") : secret "model-serving-api-tls" not found
Sep 17 00:40:32.112249 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.112221    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64fw5\" (UniqueName: \"kubernetes.io/projected/35441bf9-fae9-40d9-90a4-443a82ef0921-kube-api-access-64fw5\") pod \"model-serving-api-54d89bd5fc-hk8l5\" (UID: \"35441bf9-fae9-40d9-90a4-443a82ef0921\") " pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:32.340551 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.340512    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:32.367296 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.367266    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h5kxq"]
Sep 17 00:40:32.369485 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:40:32.369461    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009e1fc3_a69e_47c3_924a_ab5adc7066f5.slice/crio-71584a9eba5f42327f8a94fe26fa025c8965b21436c78cc763eaa54ae05be614 WatchSource:0}: Error finding container 71584a9eba5f42327f8a94fe26fa025c8965b21436c78cc763eaa54ae05be614: Status 404 returned error can't find the container with id 71584a9eba5f42327f8a94fe26fa025c8965b21436c78cc763eaa54ae05be614
Sep 17 00:40:32.606760 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.606731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35441bf9-fae9-40d9-90a4-443a82ef0921-tls-certs\") pod \"model-serving-api-54d89bd5fc-hk8l5\" (UID: \"35441bf9-fae9-40d9-90a4-443a82ef0921\") " pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:32.609242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.609225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35441bf9-fae9-40d9-90a4-443a82ef0921-tls-certs\") pod \"model-serving-api-54d89bd5fc-hk8l5\" (UID: \"35441bf9-fae9-40d9-90a4-443a82ef0921\") " pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:32.758050 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.757954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:32.863250 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:32.863217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq" event={"ID":"009e1fc3-a69e-47c3-924a-ab5adc7066f5","Type":"ContainerStarted","Data":"71584a9eba5f42327f8a94fe26fa025c8965b21436c78cc763eaa54ae05be614"}
Sep 17 00:40:33.029608 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:33.029562    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hk8l5"]
Sep 17 00:40:33.031721 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:40:33.031692    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35441bf9_fae9_40d9_90a4_443a82ef0921.slice/crio-97be6e35762c372db58d21b68c7c1ded61487e0cb362f5583ade521ab0ff695d WatchSource:0}: Error finding container 97be6e35762c372db58d21b68c7c1ded61487e0cb362f5583ade521ab0ff695d: Status 404 returned error can't find the container with id 97be6e35762c372db58d21b68c7c1ded61487e0cb362f5583ade521ab0ff695d
Sep 17 00:40:33.873646 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:33.873604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5" event={"ID":"35441bf9-fae9-40d9-90a4-443a82ef0921","Type":"ContainerStarted","Data":"97be6e35762c372db58d21b68c7c1ded61487e0cb362f5583ade521ab0ff695d"}
Sep 17 00:40:35.432005 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:35.431969    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"]
Sep 17 00:40:38.112552 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.112514    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:40:38.112820 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.112602    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:40:38.112820 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.112644    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:40:38.899269 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.899232    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5" event={"ID":"35441bf9-fae9-40d9-90a4-443a82ef0921","Type":"ContainerStarted","Data":"82732c62b9b06c662a5a3b0741bb09c18bca1bcb75ce27b7f63d6a77a4b4b538"}
Sep 17 00:40:38.899478 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.899338    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:40:38.900749 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.900726    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j" event={"ID":"730a7423-3f23-4b9a-add0-9c5e3aa6d41e","Type":"ContainerStarted","Data":"f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d"}
Sep 17 00:40:38.900853 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.900799    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j" podUID="730a7423-3f23-4b9a-add0-9c5e3aa6d41e" containerName="istio-proxy" containerID="cri-o://f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d" gracePeriod=30
Sep 17 00:40:38.902418 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.902401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq" event={"ID":"009e1fc3-a69e-47c3-924a-ab5adc7066f5","Type":"ContainerStarted","Data":"54bdf2bd540484c19a4268dd5ee697f56328fc1d8bb82690ddb6f6e08e7616f4"}
Sep 17 00:40:38.902534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.902513    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:38.918436 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.918389    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5" podStartSLOduration=2.834498277 podStartE2EDuration="7.918374296s" totalImagesPullingTime="5.083876019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:40:38.917005331 +0000 UTC m=+634.872601673" watchObservedRunningTime="2026-09-17 00:40:38.918374296 +0000 UTC m=+634.873970636"
Sep 17 00:40:38.933687 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.933637    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq" podStartSLOduration=2.197527991 podStartE2EDuration="7.933622006s" totalImagesPullingTime="5.736094015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:40:38.93303811 +0000 UTC m=+634.888634452" watchObservedRunningTime="2026-09-17 00:40:38.933622006 +0000 UTC m=+634.889218416"
Sep 17 00:40:38.952740 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:38.952691    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j" podStartSLOduration=1.7062925340000001 podStartE2EDuration="11.95267503s" totalImagesPullingTime="10.246382496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:40:38.951261247 +0000 UTC m=+634.906857599" watchObservedRunningTime="2026-09-17 00:40:38.95267503 +0000 UTC m=+634.908271370"
Sep 17 00:40:40.620658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.620624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:40.661993 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.661958    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"]
Sep 17 00:40:40.665908 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.665885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:40.668656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.668636    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 17 00:40:40.668773 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.668644    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lrdrc\""
Sep 17 00:40:40.673324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.673303    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"]
Sep 17 00:40:40.677305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.677288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:40.677715 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.677694    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"]
Sep 17 00:40:40.679621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.679602    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 17 00:40:40.679621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.679612    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2tqx8\""
Sep 17 00:40:40.694335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.694311    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"]
Sep 17 00:40:40.707810 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.707785    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:40.785008 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.784956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/490c1f14-ec34-405a-aa6f-aef946f1d204-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccv7w\" (UID: \"490c1f14-ec34-405a-aa6f-aef946f1d204\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:40.785204 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.785013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrrh\" (UniqueName: \"kubernetes.io/projected/490c1f14-ec34-405a-aa6f-aef946f1d204-kube-api-access-mjrrh\") pod \"kserve-controller-manager-d6f4b65b8-ccv7w\" (UID: \"490c1f14-ec34-405a-aa6f-aef946f1d204\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:40.785204 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.785126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48mv\" (UniqueName: \"kubernetes.io/projected/2fd22045-cb5b-4fd1-bcea-47e592e0abe1-kube-api-access-h48mv\") pod \"llmisvc-controller-manager-777c6bd7d5-kzgnq\" (UID: \"2fd22045-cb5b-4fd1-bcea-47e592e0abe1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:40.785325 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.785238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fd22045-cb5b-4fd1-bcea-47e592e0abe1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kzgnq\" (UID: \"2fd22045-cb5b-4fd1-bcea-47e592e0abe1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:40.886573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.886473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/490c1f14-ec34-405a-aa6f-aef946f1d204-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccv7w\" (UID: \"490c1f14-ec34-405a-aa6f-aef946f1d204\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:40.886573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.886511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrrh\" (UniqueName: \"kubernetes.io/projected/490c1f14-ec34-405a-aa6f-aef946f1d204-kube-api-access-mjrrh\") pod \"kserve-controller-manager-d6f4b65b8-ccv7w\" (UID: \"490c1f14-ec34-405a-aa6f-aef946f1d204\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:40.886573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.886537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48mv\" (UniqueName: \"kubernetes.io/projected/2fd22045-cb5b-4fd1-bcea-47e592e0abe1-kube-api-access-h48mv\") pod \"llmisvc-controller-manager-777c6bd7d5-kzgnq\" (UID: \"2fd22045-cb5b-4fd1-bcea-47e592e0abe1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:40.886573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.886575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fd22045-cb5b-4fd1-bcea-47e592e0abe1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kzgnq\" (UID: \"2fd22045-cb5b-4fd1-bcea-47e592e0abe1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:40.886900 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:40:40.886644    2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 00:40:40.886900 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:40:40.886682    2576 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 00:40:40.886900 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:40:40.886737    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd22045-cb5b-4fd1-bcea-47e592e0abe1-cert podName:2fd22045-cb5b-4fd1-bcea-47e592e0abe1 nodeName:}" failed. No retries permitted until 2026-09-17 00:40:41.386717964 +0000 UTC m=+637.342314284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2fd22045-cb5b-4fd1-bcea-47e592e0abe1-cert") pod "llmisvc-controller-manager-777c6bd7d5-kzgnq" (UID: "2fd22045-cb5b-4fd1-bcea-47e592e0abe1") : secret "llmisvc-webhook-server-cert" not found
Sep 17 00:40:40.886900 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:40:40.886750    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/490c1f14-ec34-405a-aa6f-aef946f1d204-cert podName:490c1f14-ec34-405a-aa6f-aef946f1d204 nodeName:}" failed. No retries permitted until 2026-09-17 00:40:41.38674437 +0000 UTC m=+637.342340689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/490c1f14-ec34-405a-aa6f-aef946f1d204-cert") pod "kserve-controller-manager-d6f4b65b8-ccv7w" (UID: "490c1f14-ec34-405a-aa6f-aef946f1d204") : secret "kserve-webhook-server-cert" not found
Sep 17 00:40:40.901134 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.901103    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrrh\" (UniqueName: \"kubernetes.io/projected/490c1f14-ec34-405a-aa6f-aef946f1d204-kube-api-access-mjrrh\") pod \"kserve-controller-manager-d6f4b65b8-ccv7w\" (UID: \"490c1f14-ec34-405a-aa6f-aef946f1d204\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:40.901272 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.901134    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48mv\" (UniqueName: \"kubernetes.io/projected/2fd22045-cb5b-4fd1-bcea-47e592e0abe1-kube-api-access-h48mv\") pod \"llmisvc-controller-manager-777c6bd7d5-kzgnq\" (UID: \"2fd22045-cb5b-4fd1-bcea-47e592e0abe1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:40.966343 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:40.966305    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:41.114262 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.114222    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:41.390125 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.390087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/490c1f14-ec34-405a-aa6f-aef946f1d204-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccv7w\" (UID: \"490c1f14-ec34-405a-aa6f-aef946f1d204\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:41.390348 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.390202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fd22045-cb5b-4fd1-bcea-47e592e0abe1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kzgnq\" (UID: \"2fd22045-cb5b-4fd1-bcea-47e592e0abe1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:41.392719 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.392696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fd22045-cb5b-4fd1-bcea-47e592e0abe1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kzgnq\" (UID: \"2fd22045-cb5b-4fd1-bcea-47e592e0abe1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:41.392844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.392742    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/490c1f14-ec34-405a-aa6f-aef946f1d204-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccv7w\" (UID: \"490c1f14-ec34-405a-aa6f-aef946f1d204\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:41.576798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.576756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:41.588714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.588687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:41.861342 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.861298    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"]
Sep 17 00:40:41.863419 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:40:41.863390    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490c1f14_ec34_405a_aa6f_aef946f1d204.slice/crio-854c760c1662226961b8b9756b0851ed77acfdccc579d4dc0872f5833ce2d783 WatchSource:0}: Error finding container 854c760c1662226961b8b9756b0851ed77acfdccc579d4dc0872f5833ce2d783: Status 404 returned error can't find the container with id 854c760c1662226961b8b9756b0851ed77acfdccc579d4dc0872f5833ce2d783
Sep 17 00:40:41.876429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.876404    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"]
Sep 17 00:40:41.879705 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:40:41.879682    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2fd22045_cb5b_4fd1_bcea_47e592e0abe1.slice/crio-1e59c5c2d8b4ace9be058135059325f0a87b31c53d45f83b3f5eee0116c2dbb5 WatchSource:0}: Error finding container 1e59c5c2d8b4ace9be058135059325f0a87b31c53d45f83b3f5eee0116c2dbb5: Status 404 returned error can't find the container with id 1e59c5c2d8b4ace9be058135059325f0a87b31c53d45f83b3f5eee0116c2dbb5
Sep 17 00:40:41.898355 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.898297    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:41.918427 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.918396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq" event={"ID":"2fd22045-cb5b-4fd1-bcea-47e592e0abe1","Type":"ContainerStarted","Data":"1e59c5c2d8b4ace9be058135059325f0a87b31c53d45f83b3f5eee0116c2dbb5"}
Sep 17 00:40:41.919593 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.919573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w" event={"ID":"490c1f14-ec34-405a-aa6f-aef946f1d204","Type":"ContainerStarted","Data":"854c760c1662226961b8b9756b0851ed77acfdccc579d4dc0872f5833ce2d783"}
Sep 17 00:40:41.994908 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:41.994872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:42.095892 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:42.095862    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:40:42.589678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:42.589650    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:44.209515 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.209492    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:44.322979 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.322941    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-token\") pod \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") "
Sep 17 00:40:44.323171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.322997    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-socket\") pod \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") "
Sep 17 00:40:44.323171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323043    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-data\") pod \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") "
Sep 17 00:40:44.323171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323081    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwt7q\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-kube-api-access-lwt7q\") pod \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") "
Sep 17 00:40:44.323171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323124    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-podinfo\") pod \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") "
Sep 17 00:40:44.323379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323189    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istiod-ca-cert\") pod \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") "
Sep 17 00:40:44.323379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323214    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-certs\") pod \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") "
Sep 17 00:40:44.323379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323241    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-envoy\") pod \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") "
Sep 17 00:40:44.323379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323271    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-credential-socket\") pod \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\" (UID: \"730a7423-3f23-4b9a-add0-9c5e3aa6d41e\") "
Sep 17 00:40:44.323379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323345    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "730a7423-3f23-4b9a-add0-9c5e3aa6d41e" (UID: "730a7423-3f23-4b9a-add0-9c5e3aa6d41e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:44.323612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323391    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-data" (OuterVolumeSpecName: "istio-data") pod "730a7423-3f23-4b9a-add0-9c5e3aa6d41e" (UID: "730a7423-3f23-4b9a-add0-9c5e3aa6d41e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:44.323612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323544    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "730a7423-3f23-4b9a-add0-9c5e3aa6d41e" (UID: "730a7423-3f23-4b9a-add0-9c5e3aa6d41e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:44.323612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323600    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:40:44.323924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323621    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:40:44.323924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323630    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "730a7423-3f23-4b9a-add0-9c5e3aa6d41e" (UID: "730a7423-3f23-4b9a-add0-9c5e3aa6d41e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:40:44.323924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.323873    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "730a7423-3f23-4b9a-add0-9c5e3aa6d41e" (UID: "730a7423-3f23-4b9a-add0-9c5e3aa6d41e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:44.325883 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.325832    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-token" (OuterVolumeSpecName: "istio-token") pod "730a7423-3f23-4b9a-add0-9c5e3aa6d41e" (UID: "730a7423-3f23-4b9a-add0-9c5e3aa6d41e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:40:44.326044 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.326019    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "730a7423-3f23-4b9a-add0-9c5e3aa6d41e" (UID: "730a7423-3f23-4b9a-add0-9c5e3aa6d41e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:44.326332 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.326309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-kube-api-access-lwt7q" (OuterVolumeSpecName: "kube-api-access-lwt7q") pod "730a7423-3f23-4b9a-add0-9c5e3aa6d41e" (UID: "730a7423-3f23-4b9a-add0-9c5e3aa6d41e"). InnerVolumeSpecName "kube-api-access-lwt7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:40:44.326493 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.326473    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "730a7423-3f23-4b9a-add0-9c5e3aa6d41e" (UID: "730a7423-3f23-4b9a-add0-9c5e3aa6d41e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:40:44.424462 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.424427    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwt7q\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-kube-api-access-lwt7q\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:40:44.424462 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.424456    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:40:44.424462 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.424468    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:40:44.424733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.424482    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:40:44.424733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.424495    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:40:44.424733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.424506    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:40:44.424733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.424516    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730a7423-3f23-4b9a-add0-9c5e3aa6d41e-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:40:44.936407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.936314    2576 generic.go:356] "Generic (PLEG): container finished" podID="730a7423-3f23-4b9a-add0-9c5e3aa6d41e" containerID="f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d" exitCode=0
Sep 17 00:40:44.936407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.936363    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"
Sep 17 00:40:44.936407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.936401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j" event={"ID":"730a7423-3f23-4b9a-add0-9c5e3aa6d41e","Type":"ContainerDied","Data":"f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d"}
Sep 17 00:40:44.936689 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.936450    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j" event={"ID":"730a7423-3f23-4b9a-add0-9c5e3aa6d41e","Type":"ContainerDied","Data":"6d9a9ab6265010c4b69405716558f4c67064d6c71e05b0361b0ac09145ce281e"}
Sep 17 00:40:44.936689 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.936500    2576 scope.go:117] "RemoveContainer" containerID="f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d"
Sep 17 00:40:44.938210 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.938180    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w" event={"ID":"490c1f14-ec34-405a-aa6f-aef946f1d204","Type":"ContainerStarted","Data":"48590792fb377707b3c4ebd707f67ec273b4d8ade0a84a29abc49a03b67908dd"}
Sep 17 00:40:44.938411 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.938394    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:40:44.945518 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.945502    2576 scope.go:117] "RemoveContainer" containerID="f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d"
Sep 17 00:40:44.945809 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:40:44.945763    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d\": container with ID starting with f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d not found: ID does not exist" containerID="f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d"
Sep 17 00:40:44.945809 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.945785    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d"} err="failed to get container status \"f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d\": rpc error: code = NotFound desc = could not find container \"f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d\": container with ID starting with f46e3b0c7cba709dc5bc9184c38e297a9a9378d8ec611e490ae8c1a813f8423d not found: ID does not exist"
Sep 17 00:40:44.956941 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.956899    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w" podStartSLOduration=2.707884001 podStartE2EDuration="4.956883782s" totalImagesPullingTime="2.248999781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:40:44.95525589 +0000 UTC m=+640.910852232" watchObservedRunningTime="2026-09-17 00:40:44.956883782 +0000 UTC m=+640.912480125"
Sep 17 00:40:44.967887 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.967867    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"]
Sep 17 00:40:44.971270 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:44.971250    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-dnz9j"]
Sep 17 00:40:45.944914 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:45.944872    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq" event={"ID":"2fd22045-cb5b-4fd1-bcea-47e592e0abe1","Type":"ContainerStarted","Data":"605a8bdc2172721a1f2525478459757d8c59cf400f03874005eaf7ed3fc12d09"}
Sep 17 00:40:45.945335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:45.945062    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:40:45.963369 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:45.963321    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq" podStartSLOduration=3.040923429 podStartE2EDuration="5.963306452s" totalImagesPullingTime="2.922383023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:40:45.961398679 +0000 UTC m=+641.916995021" watchObservedRunningTime="2026-09-17 00:40:45.963306452 +0000 UTC m=+641.918902789"
Sep 17 00:40:46.592721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:46.592690    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="730a7423-3f23-4b9a-add0-9c5e3aa6d41e" path="/var/lib/kubelet/pods/730a7423-3f23-4b9a-add0-9c5e3aa6d41e/volumes"
Sep 17 00:40:49.909205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:49.909170    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-h5kxq"
Sep 17 00:40:49.912382 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:40:49.912355    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-hk8l5"
Sep 17 00:41:15.951161 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:15.951120    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccv7w"
Sep 17 00:41:16.952557 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:16.952524    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kzgnq"
Sep 17 00:41:29.448511 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:29.448475    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:31.396139 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:31.396107    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:31.497469 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:31.497437    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:32.249746 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:32.249710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:32.344546 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:32.344511    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:37.422240 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:37.422207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:44.717670 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:44.717632    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:45.509954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:45.509926    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:48.381716 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:48.381672    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:48.525606 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:48.525581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:49.118356 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:49.118327    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:49.539022 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:49.538994    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:50.877075 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:50.877042    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:50.970834 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:50.970795    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:52.305493 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.305454    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"]
Sep 17 00:41:52.305868 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.305774    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730a7423-3f23-4b9a-add0-9c5e3aa6d41e" containerName="istio-proxy"
Sep 17 00:41:52.305868 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.305785    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="730a7423-3f23-4b9a-add0-9c5e3aa6d41e" containerName="istio-proxy"
Sep 17 00:41:52.305868 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.305853    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="730a7423-3f23-4b9a-add0-9c5e3aa6d41e" containerName="istio-proxy"
Sep 17 00:41:52.308964 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.308948    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:41:52.311453 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.311423    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 00:41:52.311588 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.311506    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 00:41:52.311588 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.311555    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8qhv2\""
Sep 17 00:41:52.318033 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.318012    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"]
Sep 17 00:41:52.394002 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.393968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98kz\" (UniqueName: \"kubernetes.io/projected/b041eb52-6883-4c7c-9058-26303581a371-kube-api-access-p98kz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" (UID: \"b041eb52-6883-4c7c-9058-26303581a371\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:41:52.394191 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.394010    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b041eb52-6883-4c7c-9058-26303581a371-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" (UID: \"b041eb52-6883-4c7c-9058-26303581a371\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:41:52.494988 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.494960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b041eb52-6883-4c7c-9058-26303581a371-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" (UID: \"b041eb52-6883-4c7c-9058-26303581a371\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:41:52.495132 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.495048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98kz\" (UniqueName: \"kubernetes.io/projected/b041eb52-6883-4c7c-9058-26303581a371-kube-api-access-p98kz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" (UID: \"b041eb52-6883-4c7c-9058-26303581a371\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:41:52.495334 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.495314    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b041eb52-6883-4c7c-9058-26303581a371-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" (UID: \"b041eb52-6883-4c7c-9058-26303581a371\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:41:52.508106 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.508085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98kz\" (UniqueName: \"kubernetes.io/projected/b041eb52-6883-4c7c-9058-26303581a371-kube-api-access-p98kz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" (UID: \"b041eb52-6883-4c7c-9058-26303581a371\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:41:52.620310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.620234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:41:52.899879 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.899858    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"]
Sep 17 00:41:52.902200 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:41:52.902172    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb041eb52_6883_4c7c_9058_26303581a371.slice/crio-55691a654fb8531123adeb3737c11f9e056f623c66b812fdc75c65284646418b WatchSource:0}: Error finding container 55691a654fb8531123adeb3737c11f9e056f623c66b812fdc75c65284646418b: Status 404 returned error can't find the container with id 55691a654fb8531123adeb3737c11f9e056f623c66b812fdc75c65284646418b
Sep 17 00:41:52.943094 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:52.943065    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:53.043490 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:53.043457    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:53.242035 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:53.241951    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4" event={"ID":"b041eb52-6883-4c7c-9058-26303581a371","Type":"ContainerStarted","Data":"55691a654fb8531123adeb3737c11f9e056f623c66b812fdc75c65284646418b"}
Sep 17 00:41:55.122160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:55.122098    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:56.572212 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:56.572116    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:58.131771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:58.131738    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:58.231354 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:58.231304    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:58.269098 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:58.269064    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4" event={"ID":"b041eb52-6883-4c7c-9058-26303581a371","Type":"ContainerStarted","Data":"57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379"}
Sep 17 00:41:58.269292 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:58.269208    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:41:58.289524 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:58.289473    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4" podStartSLOduration=1.640440377 podStartE2EDuration="6.289456952s" totalImagesPullingTime="4.649016575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:41:58.286989458 +0000 UTC m=+714.242585799" watchObservedRunningTime="2026-09-17 00:41:58.289456952 +0000 UTC m=+714.245053293"
Sep 17 00:41:58.519676 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:58.519584    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:41:58.640196 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:41:58.640105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:00.167522 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:00.167482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:00.472054 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:00.471964    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:04.322338 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:04.322302    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:04.423172 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:04.423119    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:04.523736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:04.523701    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:04.624871 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:04.624781    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:04.726400 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:04.726364    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:04.825370 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:04.825331    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:09.275230 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:09.275200    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:10.286164 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:10.286116    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:10.403182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:10.403119    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:11.162110 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.162075    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"]
Sep 17 00:42:11.168744 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.168718    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:11.180017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.179991    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"]
Sep 17 00:42:11.261437 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.261402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hnj\" (UniqueName: \"kubernetes.io/projected/9259ee85-6b90-4970-91b9-50c3ebaaa6db-kube-api-access-c7hnj\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kstzt\" (UID: \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:11.261621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.261462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9259ee85-6b90-4970-91b9-50c3ebaaa6db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kstzt\" (UID: \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:11.270190 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.270131    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"]
Sep 17 00:42:11.270422 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:11.270403    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-c7hnj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt" podUID="9259ee85-6b90-4970-91b9-50c3ebaaa6db"
Sep 17 00:42:11.307093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.307059    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"]
Sep 17 00:42:11.311290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.311267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"
Sep 17 00:42:11.327700 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.327673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"]
Sep 17 00:42:11.328746 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.328723    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:11.336232 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.336212    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:11.362544 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.362513    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hnj\" (UniqueName: \"kubernetes.io/projected/9259ee85-6b90-4970-91b9-50c3ebaaa6db-kube-api-access-c7hnj\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kstzt\" (UID: \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:11.362695 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.362578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9259ee85-6b90-4970-91b9-50c3ebaaa6db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kstzt\" (UID: \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:11.362938 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.362923    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9259ee85-6b90-4970-91b9-50c3ebaaa6db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kstzt\" (UID: \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:11.367432 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.367408    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"]
Sep 17 00:42:11.367648 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:11.367631    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-rbdpn], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-rbdpn]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g" podUID="ae5a755c-8ef4-47fb-9e5c-3c27582e8002"
Sep 17 00:42:11.375872 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.375845    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"]
Sep 17 00:42:11.376376 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.376360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hnj\" (UniqueName: \"kubernetes.io/projected/9259ee85-6b90-4970-91b9-50c3ebaaa6db-kube-api-access-c7hnj\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kstzt\" (UID: \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:11.386969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.386941    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"]
Sep 17 00:42:11.387120 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.387101    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4" podUID="b041eb52-6883-4c7c-9058-26303581a371" containerName="manager" containerID="cri-o://57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379" gracePeriod=2
Sep 17 00:42:11.391388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.391358    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"]
Sep 17 00:42:11.404586 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.404552    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:11.406049 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.406021    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"]
Sep 17 00:42:11.410041 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:11.409508    2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b041eb52-6883-4c7c-9058-26303581a371\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\": pods \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:11.411835 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.411815    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"]
Sep 17 00:42:11.415072 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.415049    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"]
Sep 17 00:42:11.416280 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.416258    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b041eb52-6883-4c7c-9058-26303581a371" containerName="manager"
Sep 17 00:42:11.416388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.416283    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b041eb52-6883-4c7c-9058-26303581a371" containerName="manager"
Sep 17 00:42:11.416388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.416369    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b041eb52-6883-4c7c-9058-26303581a371" containerName="manager"
Sep 17 00:42:11.420082 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.420065    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"]
Sep 17 00:42:11.420191 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.420173    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:11.428948 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.428926    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"]
Sep 17 00:42:11.439077 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.439035    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"]
Sep 17 00:42:11.442424 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:11.442394    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="b041eb52-6883-4c7c-9058-26303581a371" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:11.463089 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.463059    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9259ee85-6b90-4970-91b9-50c3ebaaa6db-extensions-socket-volume\") pod \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\" (UID: \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\") "
Sep 17 00:42:11.463256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.463183    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7hnj\" (UniqueName: \"kubernetes.io/projected/9259ee85-6b90-4970-91b9-50c3ebaaa6db-kube-api-access-c7hnj\") pod \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\" (UID: \"9259ee85-6b90-4970-91b9-50c3ebaaa6db\") "
Sep 17 00:42:11.463365 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.463337    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9259ee85-6b90-4970-91b9-50c3ebaaa6db-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9259ee85-6b90-4970-91b9-50c3ebaaa6db" (UID: "9259ee85-6b90-4970-91b9-50c3ebaaa6db"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:42:11.463421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.463344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmlkh\" (UniqueName: \"kubernetes.io/projected/90330fe6-2e01-4c2c-a199-723ef4f20613-kube-api-access-xmlkh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ln7xl\" (UID: \"90330fe6-2e01-4c2c-a199-723ef4f20613\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:11.463511 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.463494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90330fe6-2e01-4c2c-a199-723ef4f20613-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ln7xl\" (UID: \"90330fe6-2e01-4c2c-a199-723ef4f20613\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:11.463582 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.463569    2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9259ee85-6b90-4970-91b9-50c3ebaaa6db-extensions-socket-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:42:11.465448 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.465427    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9259ee85-6b90-4970-91b9-50c3ebaaa6db-kube-api-access-c7hnj" (OuterVolumeSpecName: "kube-api-access-c7hnj") pod "9259ee85-6b90-4970-91b9-50c3ebaaa6db" (UID: "9259ee85-6b90-4970-91b9-50c3ebaaa6db"). InnerVolumeSpecName "kube-api-access-c7hnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:42:11.486203 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:11.486164    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="b041eb52-6883-4c7c-9058-26303581a371" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:11.564563 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.564529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkxv\" (UniqueName: \"kubernetes.io/projected/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-kube-api-access-9zkxv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wznng\" (UID: \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:11.564733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.564588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90330fe6-2e01-4c2c-a199-723ef4f20613-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ln7xl\" (UID: \"90330fe6-2e01-4c2c-a199-723ef4f20613\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:11.564733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.564645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wznng\" (UID: \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:11.564733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.564692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmlkh\" (UniqueName: \"kubernetes.io/projected/90330fe6-2e01-4c2c-a199-723ef4f20613-kube-api-access-xmlkh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ln7xl\" (UID: \"90330fe6-2e01-4c2c-a199-723ef4f20613\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:11.564733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.564729    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7hnj\" (UniqueName: \"kubernetes.io/projected/9259ee85-6b90-4970-91b9-50c3ebaaa6db-kube-api-access-c7hnj\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:42:11.564933 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.564913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90330fe6-2e01-4c2c-a199-723ef4f20613-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ln7xl\" (UID: \"90330fe6-2e01-4c2c-a199-723ef4f20613\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:11.576298 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.576264    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmlkh\" (UniqueName: \"kubernetes.io/projected/90330fe6-2e01-4c2c-a199-723ef4f20613-kube-api-access-xmlkh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ln7xl\" (UID: \"90330fe6-2e01-4c2c-a199-723ef4f20613\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:11.635697 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.635665    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:11.637895 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:11.637863    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="b041eb52-6883-4c7c-9058-26303581a371" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:11.665900 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.665830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wznng\" (UID: \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:11.666053 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.666019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkxv\" (UniqueName: \"kubernetes.io/projected/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-kube-api-access-9zkxv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wznng\" (UID: \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:11.666277 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.666257    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wznng\" (UID: \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:11.675451 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.675432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkxv\" (UniqueName: \"kubernetes.io/projected/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-kube-api-access-9zkxv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wznng\" (UID: \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:11.767029 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.766994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:11.767209 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.767056    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p98kz\" (UniqueName: \"kubernetes.io/projected/b041eb52-6883-4c7c-9058-26303581a371-kube-api-access-p98kz\") pod \"b041eb52-6883-4c7c-9058-26303581a371\" (UID: \"b041eb52-6883-4c7c-9058-26303581a371\") "
Sep 17 00:42:11.767209 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.767109    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b041eb52-6883-4c7c-9058-26303581a371-extensions-socket-volume\") pod \"b041eb52-6883-4c7c-9058-26303581a371\" (UID: \"b041eb52-6883-4c7c-9058-26303581a371\") "
Sep 17 00:42:11.767495 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.767470    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b041eb52-6883-4c7c-9058-26303581a371-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b041eb52-6883-4c7c-9058-26303581a371" (UID: "b041eb52-6883-4c7c-9058-26303581a371"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:42:11.769325 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.769305    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b041eb52-6883-4c7c-9058-26303581a371-kube-api-access-p98kz" (OuterVolumeSpecName: "kube-api-access-p98kz") pod "b041eb52-6883-4c7c-9058-26303581a371" (UID: "b041eb52-6883-4c7c-9058-26303581a371"). InnerVolumeSpecName "kube-api-access-p98kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:42:11.779473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.779454    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:11.867784 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.867758    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p98kz\" (UniqueName: \"kubernetes.io/projected/b041eb52-6883-4c7c-9058-26303581a371-kube-api-access-p98kz\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:42:11.867784 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:11.867783    2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b041eb52-6883-4c7c-9058-26303581a371-extensions-socket-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:42:12.049805 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.049671    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"]
Sep 17 00:42:12.053029 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:42:12.052990    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90330fe6_2e01_4c2c_a199_723ef4f20613.slice/crio-e7adda8de000bae78bcdb77ae737284d041e85dd77e0c2834804aaef5203d827 WatchSource:0}: Error finding container e7adda8de000bae78bcdb77ae737284d041e85dd77e0c2834804aaef5203d827: Status 404 returned error can't find the container with id e7adda8de000bae78bcdb77ae737284d041e85dd77e0c2834804aaef5203d827
Sep 17 00:42:12.066109 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.066083    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"]
Sep 17 00:42:12.068915 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:42:12.068890    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee61b23c_4e3d_4ed5_982c_cbb8b7adeb64.slice/crio-36b5f3e95027b71b0a9d1ab1a2f78ddaf7ee27ab2e0d39781a35d95a0af90abc WatchSource:0}: Error finding container 36b5f3e95027b71b0a9d1ab1a2f78ddaf7ee27ab2e0d39781a35d95a0af90abc: Status 404 returned error can't find the container with id 36b5f3e95027b71b0a9d1ab1a2f78ddaf7ee27ab2e0d39781a35d95a0af90abc
Sep 17 00:42:12.334825 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.334740    2576 generic.go:356] "Generic (PLEG): container finished" podID="b041eb52-6883-4c7c-9058-26303581a371" containerID="57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379" exitCode=0
Sep 17 00:42:12.334825 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.334789    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:12.335343 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.334849    2576 scope.go:117] "RemoveContainer" containerID="57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379"
Sep 17 00:42:12.337021 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.336979    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng" event={"ID":"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64","Type":"ContainerStarted","Data":"cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e"}
Sep 17 00:42:12.337177 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.337037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng" event={"ID":"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64","Type":"ContainerStarted","Data":"36b5f3e95027b71b0a9d1ab1a2f78ddaf7ee27ab2e0d39781a35d95a0af90abc"}
Sep 17 00:42:12.337177 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.337081    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:12.337308 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:12.337284    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="b041eb52-6883-4c7c-9058-26303581a371" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:12.339268 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:12.339201    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="b041eb52-6883-4c7c-9058-26303581a371" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:12.339411 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.339308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"
Sep 17 00:42:12.339411 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.339321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl" event={"ID":"90330fe6-2e01-4c2c-a199-723ef4f20613","Type":"ContainerStarted","Data":"23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256"}
Sep 17 00:42:12.339411 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.339357    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl" event={"ID":"90330fe6-2e01-4c2c-a199-723ef4f20613","Type":"ContainerStarted","Data":"e7adda8de000bae78bcdb77ae737284d041e85dd77e0c2834804aaef5203d827"}
Sep 17 00:42:12.339585 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.339477    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:12.345211 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.345193    2576 scope.go:117] "RemoveContainer" containerID="57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379"
Sep 17 00:42:12.345525 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:12.345505    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379\": container with ID starting with 57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379 not found: ID does not exist" containerID="57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379"
Sep 17 00:42:12.345610 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.345536    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379"} err="failed to get container status \"57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379\": rpc error: code = NotFound desc = could not find container \"57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379\": container with ID starting with 57c67b7e2319950c4a02270eb86c4f38a8887765b62218340c00c403bf09a379 not found: ID does not exist"
Sep 17 00:42:12.349864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.349846    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"
Sep 17 00:42:12.363721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.363686    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng" podStartSLOduration=1.363675072 podStartE2EDuration="1.363675072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:12.363436485 +0000 UTC m=+728.319032838" watchObservedRunningTime="2026-09-17 00:42:12.363675072 +0000 UTC m=+728.319271412"
Sep 17 00:42:12.365632 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:12.365612    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kqm4\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="b041eb52-6883-4c7c-9058-26303581a371" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kqm4"
Sep 17 00:42:12.367578 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:12.367561    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-kstzt\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="9259ee85-6b90-4970-91b9-50c3ebaaa6db" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kstzt"
Sep 17 00:42:12.369372 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:12.369356    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-lqs2g\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="ae5a755c-8ef4-47fb-9e5c-3c27582e8002" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"
Sep 17 00:42:12.389765 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.389716    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl" podStartSLOduration=1.389697649 podStartE2EDuration="1.389697649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:12.388539374 +0000 UTC m=+728.344135738" watchObservedRunningTime="2026-09-17 00:42:12.389697649 +0000 UTC m=+728.345293989"
Sep 17 00:42:12.587086 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.586999    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9259ee85-6b90-4970-91b9-50c3ebaaa6db" path="/var/lib/kubelet/pods/9259ee85-6b90-4970-91b9-50c3ebaaa6db/volumes"
Sep 17 00:42:12.587302 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.587285    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae5a755c-8ef4-47fb-9e5c-3c27582e8002" path="/var/lib/kubelet/pods/ae5a755c-8ef4-47fb-9e5c-3c27582e8002/volumes"
Sep 17 00:42:12.587526 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:12.587510    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b041eb52-6883-4c7c-9058-26303581a371" path="/var/lib/kubelet/pods/b041eb52-6883-4c7c-9058-26303581a371/volumes"
Sep 17 00:42:13.345017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:13.344984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"
Sep 17 00:42:13.345550 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:13.345110    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:13.349350 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:13.349324    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-lqs2g\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="ae5a755c-8ef4-47fb-9e5c-3c27582e8002" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"
Sep 17 00:42:14.585867 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:14.585834    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-lqs2g\" is forbidden: User \"system:node:ip-10-0-143-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-51.ec2.internal' and this object" podUID="ae5a755c-8ef4-47fb-9e5c-3c27582e8002" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lqs2g"
Sep 17 00:42:14.935840 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:14.935749    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:15.041419 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:15.041387    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:18.729163 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:18.729113    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:18.829509 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:18.829479    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:19.328905 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:19.328866    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:19.389645 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:19.389602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:23.347286 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:23.347258    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:23.730925 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:23.730838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:23.782239 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:23.782205    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:24.138780 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.138745    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:24.240123 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.240095    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:24.351291 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.351263    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 00:42:24.398241 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.398156    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"]
Sep 17 00:42:24.398414 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.398336    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng" podUID="ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64" containerName="manager" containerID="cri-o://cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e" gracePeriod=10
Sep 17 00:42:24.552432 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.552403    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:24.646426 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.646391    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:24.647601 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.647584    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:24.780466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.780439    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-extensions-socket-volume\") pod \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\" (UID: \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\") "
Sep 17 00:42:24.780624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.780496    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zkxv\" (UniqueName: \"kubernetes.io/projected/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-kube-api-access-9zkxv\") pod \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\" (UID: \"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64\") "
Sep 17 00:42:24.780791 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.780770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64" (UID: "ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:42:24.782642 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.782620    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-kube-api-access-9zkxv" (OuterVolumeSpecName: "kube-api-access-9zkxv") pod "ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64" (UID: "ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64"). InnerVolumeSpecName "kube-api-access-9zkxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:42:24.881349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.881316    2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-extensions-socket-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:42:24.881349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:24.881342    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zkxv\" (UniqueName: \"kubernetes.io/projected/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64-kube-api-access-9zkxv\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:42:25.404472 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:25.404439    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64" containerID="cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e" exitCode=0
Sep 17 00:42:25.404891 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:25.404491    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"
Sep 17 00:42:25.404891 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:25.404527    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng" event={"ID":"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64","Type":"ContainerDied","Data":"cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e"}
Sep 17 00:42:25.404891 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:25.404608    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng" event={"ID":"ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64","Type":"ContainerDied","Data":"36b5f3e95027b71b0a9d1ab1a2f78ddaf7ee27ab2e0d39781a35d95a0af90abc"}
Sep 17 00:42:25.404891 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:25.404635    2576 scope.go:117] "RemoveContainer" containerID="cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e"
Sep 17 00:42:25.413065 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:25.413045    2576 scope.go:117] "RemoveContainer" containerID="cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e"
Sep 17 00:42:25.413330 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:42:25.413310    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e\": container with ID starting with cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e not found: ID does not exist" containerID="cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e"
Sep 17 00:42:25.413401 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:25.413341    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e"} err="failed to get container status \"cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e\": rpc error: code = NotFound desc = could not find container \"cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e\": container with ID starting with cf736aca903069543296cdf3f6ea63f12368abaeffbb7fe12d5a18fec284bc5e not found: ID does not exist"
Sep 17 00:42:25.427162 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:25.427127    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"]
Sep 17 00:42:25.430435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:25.430418    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wznng"]
Sep 17 00:42:26.586810 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:26.586776    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64" path="/var/lib/kubelet/pods/ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64/volumes"
Sep 17 00:42:29.991429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:29.991391    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:30.038862 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:30.038831    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:30.269264 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:30.269168    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:30.319232 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:30.319202    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:34.743979 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:34.743947    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:34.845499 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:34.845473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:40.693427 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.693395    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"]
Sep 17 00:42:40.693919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.693899    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64" containerName="manager"
Sep 17 00:42:40.693999 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.693923    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64" containerName="manager"
Sep 17 00:42:40.694052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.694001    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee61b23c-4e3d-4ed5-982c-cbb8b7adeb64" containerName="manager"
Sep 17 00:42:40.696614 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.696594    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.698925 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.698903    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 00:42:40.699052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.699023    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cncrg\""
Sep 17 00:42:40.705831 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.705786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.706870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.706846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.707036 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.707014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d869572-a142-4660-bf5c-d5f17d425672-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.707196 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.707055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d869572-a142-4660-bf5c-d5f17d425672-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.707196 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.707079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4s94\" (UniqueName: \"kubernetes.io/projected/7d869572-a142-4660-bf5c-d5f17d425672-kube-api-access-s4s94\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.707196 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.707114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.707196 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.707193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.707421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.707217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d869572-a142-4660-bf5c-d5f17d425672-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.707421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.707253    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.709088 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.709067    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"]
Sep 17 00:42:40.731837 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.731807    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:40.807739 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.807690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.807739 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.807746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808008 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.807861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d869572-a142-4660-bf5c-d5f17d425672-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808008 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.807909    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d869572-a142-4660-bf5c-d5f17d425672-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808008 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.807933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4s94\" (UniqueName: \"kubernetes.io/projected/7d869572-a142-4660-bf5c-d5f17d425672-kube-api-access-s4s94\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808008 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.807968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808259 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.808017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808259 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.808135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808259 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.808185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d869572-a142-4660-bf5c-d5f17d425672-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808259 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.808222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808496 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.808377    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808496 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.808473    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808613 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.808563    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.808729 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.808713    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d869572-a142-4660-bf5c-d5f17d425672-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.810653 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.810628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d869572-a142-4660-bf5c-d5f17d425672-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.810827 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.810805    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d869572-a142-4660-bf5c-d5f17d425672-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.815613 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.815582    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4s94\" (UniqueName: \"kubernetes.io/projected/7d869572-a142-4660-bf5c-d5f17d425672-kube-api-access-s4s94\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:40.815695 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:40.815652    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d869572-a142-4660-bf5c-d5f17d425672-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-dgjf4\" (UID: \"7d869572-a142-4660-bf5c-d5f17d425672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:41.009986 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:41.009893    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:41.185699 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:41.185661    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:41.284981 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:41.284956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"]
Sep 17 00:42:41.287123 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:42:41.287085    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d869572_a142_4660_bf5c_d5f17d425672.slice/crio-61ac45cb8276f642d2abf6b61f8fc59ceba52e9d764600cb1f07ed451ecb8778 WatchSource:0}: Error finding container 61ac45cb8276f642d2abf6b61f8fc59ceba52e9d764600cb1f07ed451ecb8778: Status 404 returned error can't find the container with id 61ac45cb8276f642d2abf6b61f8fc59ceba52e9d764600cb1f07ed451ecb8778
Sep 17 00:42:41.289186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:41.289121    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:42:41.289291 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:41.289235    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:42:41.289291 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:41.289279    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:42:41.483014 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:41.482972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" event={"ID":"7d869572-a142-4660-bf5c-d5f17d425672","Type":"ContainerStarted","Data":"5eeaf6204bb75094159abe561b19a1989951d8b9a4c21869ce8fd378e39f110f"}
Sep 17 00:42:41.483207 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:41.483036    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" event={"ID":"7d869572-a142-4660-bf5c-d5f17d425672","Type":"ContainerStarted","Data":"61ac45cb8276f642d2abf6b61f8fc59ceba52e9d764600cb1f07ed451ecb8778"}
Sep 17 00:42:41.500861 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:41.500804    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" podStartSLOduration=1.500786197 podStartE2EDuration="1.500786197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:41.499299876 +0000 UTC m=+757.454896218" watchObservedRunningTime="2026-09-17 00:42:41.500786197 +0000 UTC m=+757.456382537"
Sep 17 00:42:42.010245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.010212    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:42.014415 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.014393    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:42.449778 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.449745    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:42.462192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.462158    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r6m7f"]
Sep 17 00:42:42.465536 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.465520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:42.469677 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.469648    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l29pz\""
Sep 17 00:42:42.469813 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.469771    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 00:42:42.474391 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.474368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r6m7f"]
Sep 17 00:42:42.490667 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.490630    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:42.494415 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.494267    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:42:42.524159 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.524106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7nr\" (UniqueName: \"kubernetes.io/projected/81a7ff92-a383-4d60-933e-c0e279d19024-kube-api-access-ng7nr\") pod \"limitador-limitador-8588749977-r6m7f\" (UID: \"81a7ff92-a383-4d60-933e-c0e279d19024\") " pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:42.524345 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.524188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81a7ff92-a383-4d60-933e-c0e279d19024-config-file\") pod \"limitador-limitador-8588749977-r6m7f\" (UID: \"81a7ff92-a383-4d60-933e-c0e279d19024\") " pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:42.556824 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.556785    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r6m7f"]
Sep 17 00:42:42.565899 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.565870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:42.616243 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.616213    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:42.625116 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.625088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7nr\" (UniqueName: \"kubernetes.io/projected/81a7ff92-a383-4d60-933e-c0e279d19024-kube-api-access-ng7nr\") pod \"limitador-limitador-8588749977-r6m7f\" (UID: \"81a7ff92-a383-4d60-933e-c0e279d19024\") " pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:42.625296 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.625131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81a7ff92-a383-4d60-933e-c0e279d19024-config-file\") pod \"limitador-limitador-8588749977-r6m7f\" (UID: \"81a7ff92-a383-4d60-933e-c0e279d19024\") " pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:42.625718 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.625700    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81a7ff92-a383-4d60-933e-c0e279d19024-config-file\") pod \"limitador-limitador-8588749977-r6m7f\" (UID: \"81a7ff92-a383-4d60-933e-c0e279d19024\") " pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:42.632598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.632576    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7nr\" (UniqueName: \"kubernetes.io/projected/81a7ff92-a383-4d60-933e-c0e279d19024-kube-api-access-ng7nr\") pod \"limitador-limitador-8588749977-r6m7f\" (UID: \"81a7ff92-a383-4d60-933e-c0e279d19024\") " pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:42.777656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:42.777623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:43.046415 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:43.046389    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r6m7f"]
Sep 17 00:42:43.049438 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:42:43.049405    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a7ff92_a383_4d60_933e_c0e279d19024.slice/crio-fd363aaf86a39acaf7fa8849aeaacd09ba6b7221d890cdefd3e1a103db9a8446 WatchSource:0}: Error finding container fd363aaf86a39acaf7fa8849aeaacd09ba6b7221d890cdefd3e1a103db9a8446: Status 404 returned error can't find the container with id fd363aaf86a39acaf7fa8849aeaacd09ba6b7221d890cdefd3e1a103db9a8446
Sep 17 00:42:43.264582 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:43.264546    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:43.337796 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:43.337706    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:42:43.495124 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:43.495092    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f" event={"ID":"81a7ff92-a383-4d60-933e-c0e279d19024","Type":"ContainerStarted","Data":"fd363aaf86a39acaf7fa8849aeaacd09ba6b7221d890cdefd3e1a103db9a8446"}
Sep 17 00:42:44.501947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:44.501897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f" event={"ID":"81a7ff92-a383-4d60-933e-c0e279d19024","Type":"ContainerStarted","Data":"606d51e24c80f6656847ec81a203a3d2d41a3d33017ba0d1587b49466d17598d"}
Sep 17 00:42:44.518772 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:44.518723    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f" podStartSLOduration=1.295362665 podStartE2EDuration="2.518708369s" totalImagesPullingTime="1.223345704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:44.516427217 +0000 UTC m=+760.472023560" watchObservedRunningTime="2026-09-17 00:42:44.518708369 +0000 UTC m=+760.474304745"
Sep 17 00:42:45.506011 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:45.505984    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:56.511292 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:56.511260    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:58.112330 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:58.112298    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r6m7f"]
Sep 17 00:42:58.112752 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:58.112483    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f" podUID="81a7ff92-a383-4d60-933e-c0e279d19024" containerName="limitador" containerID="cri-o://606d51e24c80f6656847ec81a203a3d2d41a3d33017ba0d1587b49466d17598d" gracePeriod=30
Sep 17 00:42:58.568473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:58.568436    2576 generic.go:356] "Generic (PLEG): container finished" podID="81a7ff92-a383-4d60-933e-c0e279d19024" containerID="606d51e24c80f6656847ec81a203a3d2d41a3d33017ba0d1587b49466d17598d" exitCode=0
Sep 17 00:42:58.568644 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:58.568510    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f" event={"ID":"81a7ff92-a383-4d60-933e-c0e279d19024","Type":"ContainerDied","Data":"606d51e24c80f6656847ec81a203a3d2d41a3d33017ba0d1587b49466d17598d"}
Sep 17 00:42:59.057215 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.057193    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:59.176034 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.175939    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81a7ff92-a383-4d60-933e-c0e279d19024-config-file\") pod \"81a7ff92-a383-4d60-933e-c0e279d19024\" (UID: \"81a7ff92-a383-4d60-933e-c0e279d19024\") "
Sep 17 00:42:59.176034 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.176032    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng7nr\" (UniqueName: \"kubernetes.io/projected/81a7ff92-a383-4d60-933e-c0e279d19024-kube-api-access-ng7nr\") pod \"81a7ff92-a383-4d60-933e-c0e279d19024\" (UID: \"81a7ff92-a383-4d60-933e-c0e279d19024\") "
Sep 17 00:42:59.176557 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.176392    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a7ff92-a383-4d60-933e-c0e279d19024-config-file" (OuterVolumeSpecName: "config-file") pod "81a7ff92-a383-4d60-933e-c0e279d19024" (UID: "81a7ff92-a383-4d60-933e-c0e279d19024"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:42:59.178262 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.178237    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a7ff92-a383-4d60-933e-c0e279d19024-kube-api-access-ng7nr" (OuterVolumeSpecName: "kube-api-access-ng7nr") pod "81a7ff92-a383-4d60-933e-c0e279d19024" (UID: "81a7ff92-a383-4d60-933e-c0e279d19024"). InnerVolumeSpecName "kube-api-access-ng7nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:42:59.277517 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.277479    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng7nr\" (UniqueName: \"kubernetes.io/projected/81a7ff92-a383-4d60-933e-c0e279d19024-kube-api-access-ng7nr\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:42:59.277517 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.277508    2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81a7ff92-a383-4d60-933e-c0e279d19024-config-file\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:42:59.574965 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.574935    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f"
Sep 17 00:42:59.574965 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.574950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r6m7f" event={"ID":"81a7ff92-a383-4d60-933e-c0e279d19024","Type":"ContainerDied","Data":"fd363aaf86a39acaf7fa8849aeaacd09ba6b7221d890cdefd3e1a103db9a8446"}
Sep 17 00:42:59.575246 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.575017    2576 scope.go:117] "RemoveContainer" containerID="606d51e24c80f6656847ec81a203a3d2d41a3d33017ba0d1587b49466d17598d"
Sep 17 00:42:59.594958 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.594935    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r6m7f"]
Sep 17 00:42:59.598498 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:42:59.598477    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r6m7f"]
Sep 17 00:43:00.587558 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:00.587525    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81a7ff92-a383-4d60-933e-c0e279d19024" path="/var/lib/kubelet/pods/81a7ff92-a383-4d60-933e-c0e279d19024/volumes"
Sep 17 00:43:03.656596 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:03.656555    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:03.755933 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:03.755898    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:04.071157 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.071107    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"]
Sep 17 00:43:04.071531 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.071514    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81a7ff92-a383-4d60-933e-c0e279d19024" containerName="limitador"
Sep 17 00:43:04.071617 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.071534    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="81a7ff92-a383-4d60-933e-c0e279d19024" containerName="limitador"
Sep 17 00:43:04.071671 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.071623    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="81a7ff92-a383-4d60-933e-c0e279d19024" containerName="limitador"
Sep 17 00:43:04.075969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.075948    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:04.078385 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.078357    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 17 00:43:04.078537 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.078517    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 00:43:04.078624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.078614    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fbvzd\""
Sep 17 00:43:04.079531 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.079510    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 00:43:04.082251 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.082232    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"]
Sep 17 00:43:04.106512 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.106485    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:04.218780 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.218741    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5eea5411-2230-40cc-afa4-b672b56ec1f0-data\") pod \"postgres-69fb4594cf-n7q4c\" (UID: \"5eea5411-2230-40cc-afa4-b672b56ec1f0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:04.218780 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.218783    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2gzn\" (UniqueName: \"kubernetes.io/projected/5eea5411-2230-40cc-afa4-b672b56ec1f0-kube-api-access-k2gzn\") pod \"postgres-69fb4594cf-n7q4c\" (UID: \"5eea5411-2230-40cc-afa4-b672b56ec1f0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:04.319703 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.319653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5eea5411-2230-40cc-afa4-b672b56ec1f0-data\") pod \"postgres-69fb4594cf-n7q4c\" (UID: \"5eea5411-2230-40cc-afa4-b672b56ec1f0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:04.319703 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.319708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2gzn\" (UniqueName: \"kubernetes.io/projected/5eea5411-2230-40cc-afa4-b672b56ec1f0-kube-api-access-k2gzn\") pod \"postgres-69fb4594cf-n7q4c\" (UID: \"5eea5411-2230-40cc-afa4-b672b56ec1f0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:04.320054 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.320035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5eea5411-2230-40cc-afa4-b672b56ec1f0-data\") pod \"postgres-69fb4594cf-n7q4c\" (UID: \"5eea5411-2230-40cc-afa4-b672b56ec1f0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:04.328982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.328910    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2gzn\" (UniqueName: \"kubernetes.io/projected/5eea5411-2230-40cc-afa4-b672b56ec1f0-kube-api-access-k2gzn\") pod \"postgres-69fb4594cf-n7q4c\" (UID: \"5eea5411-2230-40cc-afa4-b672b56ec1f0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:04.388800 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.388769    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:04.658573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:04.658489    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"]
Sep 17 00:43:04.660697 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:43:04.660673    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eea5411_2230_40cc_afa4_b672b56ec1f0.slice/crio-301536d68a343b946b8d10553d4500d26559cfa35f9da3e659b95a79d8321e2d WatchSource:0}: Error finding container 301536d68a343b946b8d10553d4500d26559cfa35f9da3e659b95a79d8321e2d: Status 404 returned error can't find the container with id 301536d68a343b946b8d10553d4500d26559cfa35f9da3e659b95a79d8321e2d
Sep 17 00:43:05.608731 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:05.608693    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c" event={"ID":"5eea5411-2230-40cc-afa4-b672b56ec1f0","Type":"ContainerStarted","Data":"301536d68a343b946b8d10553d4500d26559cfa35f9da3e659b95a79d8321e2d"}
Sep 17 00:43:13.497972 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:13.497944    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 17 00:43:13.656078 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:13.656040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c" event={"ID":"5eea5411-2230-40cc-afa4-b672b56ec1f0","Type":"ContainerStarted","Data":"3252ed7582e96c56348f60197e28bf91c47463f6c609c3dd711778289423dc14"}
Sep 17 00:43:13.656258 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:13.656157    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:13.670961 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:13.670916    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c" podStartSLOduration=0.837754034 podStartE2EDuration="9.670879374s" totalImagesPullingTime="8.83312534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:43:13.670233229 +0000 UTC m=+789.625829571" watchObservedRunningTime="2026-09-17 00:43:13.670879374 +0000 UTC m=+789.626475714"
Sep 17 00:43:19.689360 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:19.689331    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7q4c"
Sep 17 00:43:19.993214 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:19.993088    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:20.221237 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:20.221204    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:26.954354 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:26.954272    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:27.052253 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:27.052209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:27.103777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:27.103744    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:27.350120 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:27.350084    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:27.780204 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:27.780174    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:31.230249 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:31.230208    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:43:31.311136 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:43:31.311105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:02.749316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:02.749281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:02.850510 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:02.850477    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:04.163162 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.163105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:04.263105 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.263076    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:04.331395 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.331367    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:04.646745 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.646709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:04.750294 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.750249    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:04.840363 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.840332    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:04.842814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.842779    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"]
Sep 17 00:44:04.845922 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.845901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:04.848921 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.848895    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9bt65\""
Sep 17 00:44:04.849078 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.849061    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 17 00:44:04.859047 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.859025    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"]
Sep 17 00:44:04.925506 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.925417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbgd5\" (UniqueName: \"kubernetes.io/projected/970c5f58-40ca-44c5-9c57-86ba92cc9ec1-kube-api-access-tbgd5\") pod \"payload-pre-processing-59fbc47bf8-jfwzp\" (UID: \"970c5f58-40ca-44c5-9c57-86ba92cc9ec1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:04.925506 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:04.925459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970c5f58-40ca-44c5-9c57-86ba92cc9ec1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-jfwzp\" (UID: \"970c5f58-40ca-44c5-9c57-86ba92cc9ec1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:05.025878 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:05.025840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbgd5\" (UniqueName: \"kubernetes.io/projected/970c5f58-40ca-44c5-9c57-86ba92cc9ec1-kube-api-access-tbgd5\") pod \"payload-pre-processing-59fbc47bf8-jfwzp\" (UID: \"970c5f58-40ca-44c5-9c57-86ba92cc9ec1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:05.025878 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:05.025883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970c5f58-40ca-44c5-9c57-86ba92cc9ec1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-jfwzp\" (UID: \"970c5f58-40ca-44c5-9c57-86ba92cc9ec1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:05.026608 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:05.026579    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970c5f58-40ca-44c5-9c57-86ba92cc9ec1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-jfwzp\" (UID: \"970c5f58-40ca-44c5-9c57-86ba92cc9ec1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:05.041562 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:05.041538    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbgd5\" (UniqueName: \"kubernetes.io/projected/970c5f58-40ca-44c5-9c57-86ba92cc9ec1-kube-api-access-tbgd5\") pod \"payload-pre-processing-59fbc47bf8-jfwzp\" (UID: \"970c5f58-40ca-44c5-9c57-86ba92cc9ec1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:05.156483 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:05.156445    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:05.427886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:05.427861    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"]
Sep 17 00:44:05.429658 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:44:05.429634    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970c5f58_40ca_44c5_9c57_86ba92cc9ec1.slice/crio-e0d75fb262c5648496bd342a20277daa4183d98416ac951c41c8ed2b0d0329eb WatchSource:0}: Error finding container e0d75fb262c5648496bd342a20277daa4183d98416ac951c41c8ed2b0d0329eb: Status 404 returned error can't find the container with id e0d75fb262c5648496bd342a20277daa4183d98416ac951c41c8ed2b0d0329eb
Sep 17 00:44:05.887792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:05.887756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp" event={"ID":"970c5f58-40ca-44c5-9c57-86ba92cc9ec1","Type":"ContainerStarted","Data":"e0d75fb262c5648496bd342a20277daa4183d98416ac951c41c8ed2b0d0329eb"}
Sep 17 00:44:08.903381 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:08.903341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp" event={"ID":"970c5f58-40ca-44c5-9c57-86ba92cc9ec1","Type":"ContainerStarted","Data":"eb7ac97dc1423c0f0c90a38f04dd3b707c1a0a9335bcd5aef8f8233303ef446f"}
Sep 17 00:44:08.903804 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:08.903445    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:08.919276 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:08.919230    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp" podStartSLOduration=1.944840841 podStartE2EDuration="4.919214208s" totalImagesPullingTime="2.974373367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:44:08.917038397 +0000 UTC m=+844.872634750" watchObservedRunningTime="2026-09-17 00:44:08.919214208 +0000 UTC m=+844.874810548"
Sep 17 00:44:13.532325 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:13.532284    2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-dgjf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 00:44:13.532764 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:13.532346    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" podUID="7d869572-a142-4660-bf5c-d5f17d425672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 00:44:14.934732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:14.934694    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d869572-a142-4660-bf5c-d5f17d425672" containerID="5eeaf6204bb75094159abe561b19a1989951d8b9a4c21869ce8fd378e39f110f" exitCode=0
Sep 17 00:44:14.935205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:14.934765    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" event={"ID":"7d869572-a142-4660-bf5c-d5f17d425672","Type":"ContainerDied","Data":"5eeaf6204bb75094159abe561b19a1989951d8b9a4c21869ce8fd378e39f110f"}
Sep 17 00:44:14.935205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:14.935200    2576 scope.go:117] "RemoveContainer" containerID="5eeaf6204bb75094159abe561b19a1989951d8b9a4c21869ce8fd378e39f110f"
Sep 17 00:44:14.936189 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:14.936160    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:44:14.936253 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:14.936230    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:44:14.936301 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:14.936266    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:44:15.064301 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:15.064274    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:15.160494 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:15.160458    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:15.944559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:15.944509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" event={"ID":"7d869572-a142-4660-bf5c-d5f17d425672","Type":"ContainerStarted","Data":"17843c3ad0af2432cca8d1a09baedf31649bd61e912ef17991c4495fba251336"}
Sep 17 00:44:16.010987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:16.010957    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:44:16.011174 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:16.011073    2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-dgjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 00:44:16.011174 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:16.011127    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" podUID="7d869572-a142-4660-bf5c-d5f17d425672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 00:44:17.010428 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:17.010393    2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-dgjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 00:44:17.010900 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:17.010462    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" podUID="7d869572-a142-4660-bf5c-d5f17d425672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 00:44:18.010815 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:18.010775    2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-dgjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 00:44:18.011308 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:18.010835    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" podUID="7d869572-a142-4660-bf5c-d5f17d425672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 00:44:19.010895 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:19.010859    2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-dgjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 00:44:19.011388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:19.010922    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" podUID="7d869572-a142-4660-bf5c-d5f17d425672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 00:44:19.911176 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:19.911130    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-jfwzp"
Sep 17 00:44:20.010915 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:20.010882    2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-dgjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 00:44:20.011308 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:20.010943    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" podUID="7d869572-a142-4660-bf5c-d5f17d425672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 00:44:21.010947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:21.010913    2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-dgjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 00:44:21.011348 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:21.010983    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4" podUID="7d869572-a142-4660-bf5c-d5f17d425672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 00:44:22.013857 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:22.013830    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:44:22.014220 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:22.014104    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:44:22.014873 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:22.014855    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-dgjf4"
Sep 17 00:44:22.787319 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:22.787287    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:22.835627 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:22.835601    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:54.695734 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.695683    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"]
Sep 17 00:44:54.699390 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.699370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.701717 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.701697    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 17 00:44:54.702741 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.702721    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 00:44:54.702965 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.702731    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vlt7q\""
Sep 17 00:44:54.703075 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.702740    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 00:44:54.708312 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.708289    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"]
Sep 17 00:44:54.726618 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.726573    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:44:54.782046 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.782016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.782200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.782059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.782200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.782079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.782200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.782170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.782345 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.782213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mdw\" (UniqueName: \"kubernetes.io/projected/c0bea1a0-f99d-46fd-b850-211bdfb4d794-kube-api-access-x9mdw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.782345 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.782239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bea1a0-f99d-46fd-b850-211bdfb4d794-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.883602 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.883563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.883602 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.883604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.883844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.883632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.883844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.883662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.883844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.883697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mdw\" (UniqueName: \"kubernetes.io/projected/c0bea1a0-f99d-46fd-b850-211bdfb4d794-kube-api-access-x9mdw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.883844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.883720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bea1a0-f99d-46fd-b850-211bdfb4d794-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.884103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.884076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.884206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.884084    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.884206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.884174    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.886032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.886010    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bea1a0-f99d-46fd-b850-211bdfb4d794-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.886347 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.886329    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bea1a0-f99d-46fd-b850-211bdfb4d794-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:54.891226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:54.891208    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mdw\" (UniqueName: \"kubernetes.io/projected/c0bea1a0-f99d-46fd-b850-211bdfb4d794-kube-api-access-x9mdw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg\" (UID: \"c0bea1a0-f99d-46fd-b850-211bdfb4d794\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:55.031131 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:55.031100    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:44:55.306344 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:55.306319    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"]
Sep 17 00:44:55.308221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:44:55.308199    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bea1a0_f99d_46fd_b850_211bdfb4d794.slice/crio-0f162fb9dba4de6c99ff6193c16dff1a5ff1022cad5686445ad9af944e0247b2 WatchSource:0}: Error finding container 0f162fb9dba4de6c99ff6193c16dff1a5ff1022cad5686445ad9af944e0247b2: Status 404 returned error can't find the container with id 0f162fb9dba4de6c99ff6193c16dff1a5ff1022cad5686445ad9af944e0247b2
Sep 17 00:44:55.309546 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:55.309528    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:44:56.148992 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:56.148957    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg" event={"ID":"c0bea1a0-f99d-46fd-b850-211bdfb4d794","Type":"ContainerStarted","Data":"0f162fb9dba4de6c99ff6193c16dff1a5ff1022cad5686445ad9af944e0247b2"}
Sep 17 00:44:57.155048 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:57.155009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg" event={"ID":"c0bea1a0-f99d-46fd-b850-211bdfb4d794","Type":"ContainerStarted","Data":"11da5a7f6d42d389cf084de6daf55cbd3fbd1692e22c0b97e04c8425303a07f5"}
Sep 17 00:44:57.171757 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:44:57.171703    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg" podStartSLOduration=1.8803459820000001 podStartE2EDuration="3.17168699s" totalImagesPullingTime="1.291341008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:44:57.169928594 +0000 UTC m=+893.125524937" watchObservedRunningTime="2026-09-17 00:44:57.17168699 +0000 UTC m=+893.127283330"
Sep 17 00:45:00.135480 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:00.135445    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm"]
Sep 17 00:45:00.138788 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:00.138768    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm"
Sep 17 00:45:00.141302 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:00.141281    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:45:00.143329 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:00.143268    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm"]
Sep 17 00:45:00.151382 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:00.151363    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm"
Sep 17 00:45:00.423814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:00.423790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm"]
Sep 17 00:45:00.425589 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:45:00.425560    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb52972_2499_4419_8d3f_e02751747455.slice/crio-ba9c0c11acdad2da2e8b2dd23c9a351e9eb48a7ed21b86c8bef71d7d0ef1c6ed WatchSource:0}: Error finding container ba9c0c11acdad2da2e8b2dd23c9a351e9eb48a7ed21b86c8bef71d7d0ef1c6ed: Status 404 returned error can't find the container with id ba9c0c11acdad2da2e8b2dd23c9a351e9eb48a7ed21b86c8bef71d7d0ef1c6ed
Sep 17 00:45:01.176185 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:01.176134    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm" event={"ID":"bbb52972-2499-4419-8d3f-e02751747455","Type":"ContainerStarted","Data":"ba9c0c11acdad2da2e8b2dd23c9a351e9eb48a7ed21b86c8bef71d7d0ef1c6ed"}
Sep 17 00:45:03.187272 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:03.187234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm" event={"ID":"bbb52972-2499-4419-8d3f-e02751747455","Type":"ContainerStarted","Data":"f608067d1436d4811c7600fdf755ab4516fec6dc9fa3f019f91ff303fbe1a684"}
Sep 17 00:45:03.201559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:03.201512    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm" podStartSLOduration=0.897155357 podStartE2EDuration="3.201496869s" totalImagesPullingTime="2.304341512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:45:03.200245758 +0000 UTC m=+899.155842098" watchObservedRunningTime="2026-09-17 00:45:03.201496869 +0000 UTC m=+899.157093228"
Sep 17 00:45:04.193857 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:04.193817    2576 generic.go:356] "Generic (PLEG): container finished" podID="bbb52972-2499-4419-8d3f-e02751747455" containerID="f608067d1436d4811c7600fdf755ab4516fec6dc9fa3f019f91ff303fbe1a684" exitCode=0
Sep 17 00:45:04.194302 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:04.193911    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm" event={"ID":"bbb52972-2499-4419-8d3f-e02751747455","Type":"ContainerDied","Data":"f608067d1436d4811c7600fdf755ab4516fec6dc9fa3f019f91ff303fbe1a684"}
Sep 17 00:45:04.965841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:04.965810    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:45:04.966017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:04.965871    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:45:05.031793 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:05.031758    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:45:05.031944 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:05.031822    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:45:05.044219 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:05.044192    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:45:05.210662 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:05.210632    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg"
Sep 17 00:45:05.334426 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:05.334399    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm"
Sep 17 00:45:06.204918 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:06.204889    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm"
Sep 17 00:45:06.205095 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:06.204934    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826765-5v5nm" event={"ID":"bbb52972-2499-4419-8d3f-e02751747455","Type":"ContainerDied","Data":"ba9c0c11acdad2da2e8b2dd23c9a351e9eb48a7ed21b86c8bef71d7d0ef1c6ed"}
Sep 17 00:45:06.205095 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:06.204981    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba9c0c11acdad2da2e8b2dd23c9a351e9eb48a7ed21b86c8bef71d7d0ef1c6ed"
Sep 17 00:45:23.926568 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:23.926537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:45:39.492532 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.492496    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"]
Sep 17 00:45:39.493428 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.492877    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbb52972-2499-4419-8d3f-e02751747455" containerName="cleanup"
Sep 17 00:45:39.493428 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.492894    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbb52972-2499-4419-8d3f-e02751747455" containerName="cleanup"
Sep 17 00:45:39.493428 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.492987    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbb52972-2499-4419-8d3f-e02751747455" containerName="cleanup"
Sep 17 00:45:39.496298 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.496278    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.498756 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.498733    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 17 00:45:39.505495 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.505470    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"]
Sep 17 00:45:39.530426 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.530392    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:45:39.581868 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.581830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.582053 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.581884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.582053 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.581911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.582053 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.581934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.582216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.582075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0b28f85-aef9-47ea-8236-900bd042c1f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.582216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.582114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f28c\" (UniqueName: \"kubernetes.io/projected/a0b28f85-aef9-47ea-8236-900bd042c1f1-kube-api-access-9f28c\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.683000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.682960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.683000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.682999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.683290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.683015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.683290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.683065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0b28f85-aef9-47ea-8236-900bd042c1f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.683290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.683108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f28c\" (UniqueName: \"kubernetes.io/projected/a0b28f85-aef9-47ea-8236-900bd042c1f1-kube-api-access-9f28c\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.683290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.683167    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.683508 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.683445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.683508 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.683476    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.683889 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.683865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.685983 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.685964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0b28f85-aef9-47ea-8236-900bd042c1f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.686070 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.686054    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0b28f85-aef9-47ea-8236-900bd042c1f1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.691001 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.690978    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f28c\" (UniqueName: \"kubernetes.io/projected/a0b28f85-aef9-47ea-8236-900bd042c1f1-kube-api-access-9f28c\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7\" (UID: \"a0b28f85-aef9-47ea-8236-900bd042c1f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:39.809309 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:39.809264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:40.081108 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:40.081083    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"]
Sep 17 00:45:40.083313 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:45:40.083274    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b28f85_aef9_47ea_8236_900bd042c1f1.slice/crio-cef7895712e2e20644dc422491fd342ca212dde8942eb556e9119dce577cdcc8 WatchSource:0}: Error finding container cef7895712e2e20644dc422491fd342ca212dde8942eb556e9119dce577cdcc8: Status 404 returned error can't find the container with id cef7895712e2e20644dc422491fd342ca212dde8942eb556e9119dce577cdcc8
Sep 17 00:45:40.367953 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:40.367911    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7" event={"ID":"a0b28f85-aef9-47ea-8236-900bd042c1f1","Type":"ContainerStarted","Data":"9e110292b522c6edaff201b00cab450f00ee41e1632222aaee9bf90f2e0bf083"}
Sep 17 00:45:40.368172 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:40.367974    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7" event={"ID":"a0b28f85-aef9-47ea-8236-900bd042c1f1","Type":"ContainerStarted","Data":"cef7895712e2e20644dc422491fd342ca212dde8942eb556e9119dce577cdcc8"}
Sep 17 00:45:40.385350 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:40.385301    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7" podStartSLOduration=1.192519669 podStartE2EDuration="1.385285373s" totalImagesPullingTime="192.765704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:45:40.384083076 +0000 UTC m=+936.339679416" watchObservedRunningTime="2026-09-17 00:45:40.385285373 +0000 UTC m=+936.340881866"
Sep 17 00:45:43.027465 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:43.027433    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:45:43.125397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:43.125367    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:45:46.820475 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:46.820443    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:45:49.810325 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:49.810280    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:49.810732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:49.810442    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:49.822657 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:49.822632    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:45:50.427758 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:45:50.427725    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7"
Sep 17 00:46:07.225489 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:46:07.225458    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:46:15.225630 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:46:15.225598    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:46:31.823350 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:46:31.823272    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:46:52.095226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:46:52.095195    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:46:52.195590 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:46:52.195559    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:05.004884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:05.004850    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:05.055635 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:05.055606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:17.738229 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:17.738166    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:24.411813 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:24.411777    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:40.073621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:40.073578    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:40.174016 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:40.173978    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:41.158241 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:41.158210    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:41.257882 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:41.257850    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:56.384130 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.384048    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:56.437713 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.437683    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:56.501465 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.501434    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:56.680451 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.680367    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"]
Sep 17 00:47:56.683811 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.683792    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.687995 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.687973    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e491c5-openshift-default-dockercfg-7htf9\""
Sep 17 00:47:56.696812 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.696788    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"]
Sep 17 00:47:56.709968 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.709944    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:56.784994 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.784960    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:56.819335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819303    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"]
Sep 17 00:47:56.819648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-certs\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.819727 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-podinfo\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.819727 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istiod-ca-cert\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.819842 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-data\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.819842 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8pd\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-kube-api-access-sg8pd\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.819947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-token\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.819947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-envoy\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.819947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-socket\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.819947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.819921    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-credential-socket\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.824421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.824398    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.829515 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.829492    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f40754-openshift-default-dockercfg-4hgzx\""
Sep 17 00:47:56.837386 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.837364    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"]
Sep 17 00:47:56.862187 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.862164    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:56.921473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.921434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-data\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.921675 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.921489    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8pd\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-kube-api-access-sg8pd\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.921675 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.921533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istiod-ca-cert\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.921675 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.921561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-token\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.921675 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.921586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-credential-socket\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.921930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.921903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-envoy\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.921989 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.921974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-socket\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.922081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-credential-socket\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.922419 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922394    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-socket\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.922525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-data\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.922590 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-token\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.922646 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-certs\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.922703 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-podinfo\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.922756 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-certs\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.922756 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-podinfo\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.922859 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istiod-ca-cert\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.922859 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhw76\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-kube-api-access-rhw76\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.922859 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-envoy\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.923046 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.922868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-socket\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:56.923403 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.923383    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-credential-socket\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.923539 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.923392    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-certs\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.923689 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.923637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istiod-ca-cert\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.924079 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.924050    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-data\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.924787 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.924762    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-envoy\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.925773 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.925747    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-podinfo\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.931139 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.931068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-token\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.931261 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.931190    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8pd\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-kube-api-access-sg8pd\") pod \"e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") " pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:56.994600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:56.994552    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:57.024178 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-socket\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.024313 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istiod-ca-cert\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.024313 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-credential-socket\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.024313 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-data\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.024425 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-token\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.024425 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-certs\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.024491 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024435    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-podinfo\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.024491 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhw76\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-kube-api-access-rhw76\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.024594 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-envoy\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.024594 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.024529    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-socket\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.025201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.025065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-credential-socket\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.025201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.025167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istiod-ca-cert\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.025716 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.025475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-data\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.025716 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.025682    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-certs\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.028444 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.028420    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-envoy\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.028870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.028845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-podinfo\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.035339 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.035292    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhw76\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-kube-api-access-rhw76\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.035339 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.035304    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-token\") pod \"e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") " pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.058071 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.057903    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:57.141420 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.141384    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:57.177458 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.177415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:57.283243 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.283218    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"]
Sep 17 00:47:57.285695 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:47:57.285667    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161c2fcd_59bb_46b4_8899_45cbac6e01ff.slice/crio-33bf67f118056b0485c0c50bac3a23faec3cc08a8a2cd4143add64f4dcda752a WatchSource:0}: Error finding container 33bf67f118056b0485c0c50bac3a23faec3cc08a8a2cd4143add64f4dcda752a: Status 404 returned error can't find the container with id 33bf67f118056b0485c0c50bac3a23faec3cc08a8a2cd4143add64f4dcda752a
Sep 17 00:47:57.287752 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.287723    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:47:57.287813 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.287789    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:47:57.287851 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.287819    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:47:57.303121 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.303028    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:57.421881 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.421854    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"]
Sep 17 00:47:57.424681 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:47:57.424652    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a97aaa1_df48_41b0_a7a3_830e8bbef608.slice/crio-0082e5d0536376cd2d92a6685a2a1f5834f6db693a35d361e37277ec6bcab8ba WatchSource:0}: Error finding container 0082e5d0536376cd2d92a6685a2a1f5834f6db693a35d361e37277ec6bcab8ba: Status 404 returned error can't find the container with id 0082e5d0536376cd2d92a6685a2a1f5834f6db693a35d361e37277ec6bcab8ba
Sep 17 00:47:57.427517 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.427482    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:47:57.427630 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.427552    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:47:57.427630 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.427583    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:47:57.741105 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.740988    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:57.841870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:57.841837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:58.011456 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.011359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" event={"ID":"1a97aaa1-df48-41b0-a7a3-830e8bbef608","Type":"ContainerStarted","Data":"903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8"}
Sep 17 00:47:58.011456 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.011450    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" event={"ID":"1a97aaa1-df48-41b0-a7a3-830e8bbef608","Type":"ContainerStarted","Data":"0082e5d0536376cd2d92a6685a2a1f5834f6db693a35d361e37277ec6bcab8ba"}
Sep 17 00:47:58.013084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.013057    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" event={"ID":"161c2fcd-59bb-46b4-8899-45cbac6e01ff","Type":"ContainerStarted","Data":"f4f655c65eb369b86078520804ac7a787ff1e5c71651d7540f2ed4070c2ef1cd"}
Sep 17 00:47:58.013214 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.013101    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" event={"ID":"161c2fcd-59bb-46b4-8899-45cbac6e01ff","Type":"ContainerStarted","Data":"33bf67f118056b0485c0c50bac3a23faec3cc08a8a2cd4143add64f4dcda752a"}
Sep 17 00:47:58.032213 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.032139    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podStartSLOduration=2.032121223 podStartE2EDuration="2.032121223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:47:58.03118595 +0000 UTC m=+1073.986782293" watchObservedRunningTime="2026-09-17 00:47:58.032121223 +0000 UTC m=+1073.987717576"
Sep 17 00:47:58.052233 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.052168    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" podStartSLOduration=2.052130956 podStartE2EDuration="2.052130956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:47:58.049255827 +0000 UTC m=+1074.004852169" watchObservedRunningTime="2026-09-17 00:47:58.052130956 +0000 UTC m=+1074.007727339"
Sep 17 00:47:58.142594 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.142378    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:58.143700 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.143672    2576 patch_prober.go:28] interesting pod/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 17 00:47:58.143802 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.143731    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 17 00:47:58.187662 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.187632    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:58.995247 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:58.995211    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:59.000520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:59.000496    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:59.017847 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:59.017819    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:59.018608 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:59.018591    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:47:59.146682 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:59.146652    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:47:59.601059 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:59.601013    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:59.700323 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:59.700293    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:47:59.970009 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:47:59.969912    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:00.022490 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:00.022449    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:48:00.023546 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:00.023521    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:48:00.054573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:00.054535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:00.119081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:00.119042    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:00.765806 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:00.765688    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:00.866451 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:00.866415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:00.984748 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:00.984711    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:01.287623 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.287591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:01.377455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.377424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:01.385103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.385076    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"]
Sep 17 00:48:01.388641 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.388619    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:01.391239 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.391222    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a79f56\""
Sep 17 00:48:01.398396 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.398374    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"]
Sep 17 00:48:01.421481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.421455    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"]
Sep 17 00:48:01.424945 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.424929    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:01.427455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.427436    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a79f56-dockercfg-vhrt7\""
Sep 17 00:48:01.433157 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.433120    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"]
Sep 17 00:48:01.472661 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.472627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkr94\" (UniqueName: \"kubernetes.io/projected/e09dedfe-73ef-46b8-bf08-d037278c201c-kube-api-access-qkr94\") pod \"payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx\" (UID: \"e09dedfe-73ef-46b8-bf08-d037278c201c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:01.472834 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.472669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09dedfe-73ef-46b8-bf08-d037278c201c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx\" (UID: \"e09dedfe-73ef-46b8-bf08-d037278c201c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:01.472834 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.472760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d9df51-4813-44ca-866a-8f2b0c3b4941-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm\" (UID: \"13d9df51-4813-44ca-866a-8f2b0c3b4941\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:01.472834 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.472781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdjr6\" (UniqueName: \"kubernetes.io/projected/13d9df51-4813-44ca-866a-8f2b0c3b4941-kube-api-access-fdjr6\") pod \"payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm\" (UID: \"13d9df51-4813-44ca-866a-8f2b0c3b4941\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:01.573319 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.573224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d9df51-4813-44ca-866a-8f2b0c3b4941-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm\" (UID: \"13d9df51-4813-44ca-866a-8f2b0c3b4941\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:01.573319 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.573269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdjr6\" (UniqueName: \"kubernetes.io/projected/13d9df51-4813-44ca-866a-8f2b0c3b4941-kube-api-access-fdjr6\") pod \"payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm\" (UID: \"13d9df51-4813-44ca-866a-8f2b0c3b4941\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:01.573555 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.573371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkr94\" (UniqueName: \"kubernetes.io/projected/e09dedfe-73ef-46b8-bf08-d037278c201c-kube-api-access-qkr94\") pod \"payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx\" (UID: \"e09dedfe-73ef-46b8-bf08-d037278c201c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:01.573555 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.573408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09dedfe-73ef-46b8-bf08-d037278c201c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx\" (UID: \"e09dedfe-73ef-46b8-bf08-d037278c201c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:01.574007 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.573980    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d9df51-4813-44ca-866a-8f2b0c3b4941-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm\" (UID: \"13d9df51-4813-44ca-866a-8f2b0c3b4941\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:01.574199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.574179    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09dedfe-73ef-46b8-bf08-d037278c201c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx\" (UID: \"e09dedfe-73ef-46b8-bf08-d037278c201c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:01.581941 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.581918    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdjr6\" (UniqueName: \"kubernetes.io/projected/13d9df51-4813-44ca-866a-8f2b0c3b4941-kube-api-access-fdjr6\") pod \"payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm\" (UID: \"13d9df51-4813-44ca-866a-8f2b0c3b4941\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:01.582093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.582069    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkr94\" (UniqueName: \"kubernetes.io/projected/e09dedfe-73ef-46b8-bf08-d037278c201c-kube-api-access-qkr94\") pod \"payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx\" (UID: \"e09dedfe-73ef-46b8-bf08-d037278c201c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:01.700195 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.700154    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:01.736076 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.736047    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:01.774897 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.774774    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:01.872292 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:01.872259    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:02.000547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.000463    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"]
Sep 17 00:48:02.005220 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:48:02.005190    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d9df51_4813_44ca_866a_8f2b0c3b4941.slice/crio-b33a83869b985b961da4c472393260a08b3463cab428dfb1700b1317f9a8d9a4 WatchSource:0}: Error finding container b33a83869b985b961da4c472393260a08b3463cab428dfb1700b1317f9a8d9a4: Status 404 returned error can't find the container with id b33a83869b985b961da4c472393260a08b3463cab428dfb1700b1317f9a8d9a4
Sep 17 00:48:02.033679 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.033652    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm" event={"ID":"13d9df51-4813-44ca-866a-8f2b0c3b4941","Type":"ContainerStarted","Data":"b33a83869b985b961da4c472393260a08b3463cab428dfb1700b1317f9a8d9a4"}
Sep 17 00:48:02.071705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.071649    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"]
Sep 17 00:48:02.073658 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:48:02.073522    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09dedfe_73ef_46b8_bf08_d037278c201c.slice/crio-38edbe1caa6bb88c8968070df5f9bcd26529a3835f887559fd3f9ab08b4041ff WatchSource:0}: Error finding container 38edbe1caa6bb88c8968070df5f9bcd26529a3835f887559fd3f9ab08b4041ff: Status 404 returned error can't find the container with id 38edbe1caa6bb88c8968070df5f9bcd26529a3835f887559fd3f9ab08b4041ff
Sep 17 00:48:02.163787 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.163737    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:02.274696 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.274580    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:02.360338 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.360248    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:02.408717 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.407845    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"]
Sep 17 00:48:02.411557 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.411536    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:02.413939 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.413909    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e491c5-dockercfg-tl6fh\""
Sep 17 00:48:02.413939 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.413928    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e491c5\""
Sep 17 00:48:02.419179 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.419157    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"]
Sep 17 00:48:02.481511 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.481300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6\" (UID: \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:02.481511 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.481353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fhk5\" (UniqueName: \"kubernetes.io/projected/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-kube-api-access-2fhk5\") pod \"payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6\" (UID: \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:02.582845 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.582812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6\" (UID: \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:02.583021 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.582850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fhk5\" (UniqueName: \"kubernetes.io/projected/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-kube-api-access-2fhk5\") pod \"payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6\" (UID: \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:02.583803 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.583777    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6\" (UID: \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:02.590352 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.590329    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fhk5\" (UniqueName: \"kubernetes.io/projected/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-kube-api-access-2fhk5\") pod \"payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6\" (UID: \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:02.656599 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.656518    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:02.725098 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.725064    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:02.757839 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.757809    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:02.924455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:02.924367    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:03.011090 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.011067    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"]
Sep 17 00:48:03.013507 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:48:03.013474    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ad4838_8ed1_4a6b_b202_9964ca6e3c7f.slice/crio-d93c23d83fc36af9c3db239ab1b62ba1ed8bec7edf52402ef4081a41cdd56782 WatchSource:0}: Error finding container d93c23d83fc36af9c3db239ab1b62ba1ed8bec7edf52402ef4081a41cdd56782: Status 404 returned error can't find the container with id d93c23d83fc36af9c3db239ab1b62ba1ed8bec7edf52402ef4081a41cdd56782
Sep 17 00:48:03.041231 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.041164    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6" event={"ID":"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f","Type":"ContainerStarted","Data":"d93c23d83fc36af9c3db239ab1b62ba1ed8bec7edf52402ef4081a41cdd56782"}
Sep 17 00:48:03.044468 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.043727    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx" event={"ID":"e09dedfe-73ef-46b8-bf08-d037278c201c","Type":"ContainerStarted","Data":"2ca321f051f8de4f6b08040eac3ce211f344c7b2baefb3001724d4c164203de8"}
Sep 17 00:48:03.044468 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.043782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx" event={"ID":"e09dedfe-73ef-46b8-bf08-d037278c201c","Type":"ContainerStarted","Data":"38edbe1caa6bb88c8968070df5f9bcd26529a3835f887559fd3f9ab08b4041ff"}
Sep 17 00:48:03.044468 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.044046    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:03.045965 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.045936    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm" event={"ID":"13d9df51-4813-44ca-866a-8f2b0c3b4941","Type":"ContainerStarted","Data":"fa4bafc2e3b6b108fcb472575a250a99f0cc08fcc743cdc3f384aca2f671db96"}
Sep 17 00:48:03.046112 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.046066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:03.062632 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.062582    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx" podStartSLOduration=2.062558562 podStartE2EDuration="2.062558562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:03.058974848 +0000 UTC m=+1079.014571194" watchObservedRunningTime="2026-09-17 00:48:03.062558562 +0000 UTC m=+1079.018154904"
Sep 17 00:48:03.077347 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.077297    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm" podStartSLOduration=2.077277429 podStartE2EDuration="2.077277429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:03.074537109 +0000 UTC m=+1079.030133452" watchObservedRunningTime="2026-09-17 00:48:03.077277429 +0000 UTC m=+1079.032873773"
Sep 17 00:48:03.138336 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.138278    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:03.232259 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.232176    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:03.271884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.271845    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"]
Sep 17 00:48:03.275522 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.275501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:03.277830 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.277810    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f40754-dockercfg-d8z56\""
Sep 17 00:48:03.277958 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.277852    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f40754\""
Sep 17 00:48:03.285108 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.285025    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"]
Sep 17 00:48:03.290410 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.290384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba12a105-636d-4d55-b5b7-b15ca90b5453-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4\" (UID: \"ba12a105-636d-4d55-b5b7-b15ca90b5453\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:03.290536 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.290416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbh78\" (UniqueName: \"kubernetes.io/projected/ba12a105-636d-4d55-b5b7-b15ca90b5453-kube-api-access-kbh78\") pod \"payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4\" (UID: \"ba12a105-636d-4d55-b5b7-b15ca90b5453\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:03.391803 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.391762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba12a105-636d-4d55-b5b7-b15ca90b5453-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4\" (UID: \"ba12a105-636d-4d55-b5b7-b15ca90b5453\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:03.391803 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.391808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbh78\" (UniqueName: \"kubernetes.io/projected/ba12a105-636d-4d55-b5b7-b15ca90b5453-kube-api-access-kbh78\") pod \"payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4\" (UID: \"ba12a105-636d-4d55-b5b7-b15ca90b5453\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:03.392605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.392584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba12a105-636d-4d55-b5b7-b15ca90b5453-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4\" (UID: \"ba12a105-636d-4d55-b5b7-b15ca90b5453\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:03.399680 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.399642    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbh78\" (UniqueName: \"kubernetes.io/projected/ba12a105-636d-4d55-b5b7-b15ca90b5453-kube-api-access-kbh78\") pod \"payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4\" (UID: \"ba12a105-636d-4d55-b5b7-b15ca90b5453\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:03.588393 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.588366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:03.656684 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.656654    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:03.869056 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:03.869033    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"]
Sep 17 00:48:03.870880 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:48:03.870843    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba12a105_636d_4d55_b5b7_b15ca90b5453.slice/crio-8415044902d64c6371fa3c43a787b2424ce260a38d2ba691d54051946d321a9d WatchSource:0}: Error finding container 8415044902d64c6371fa3c43a787b2424ce260a38d2ba691d54051946d321a9d: Status 404 returned error can't find the container with id 8415044902d64c6371fa3c43a787b2424ce260a38d2ba691d54051946d321a9d
Sep 17 00:48:04.054579 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.054534    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6" event={"ID":"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f","Type":"ContainerStarted","Data":"70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483"}
Sep 17 00:48:04.054774 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.054668    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:04.056514 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.056452    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4" event={"ID":"ba12a105-636d-4d55-b5b7-b15ca90b5453","Type":"ContainerStarted","Data":"1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73"}
Sep 17 00:48:04.056514 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.056505    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4" event={"ID":"ba12a105-636d-4d55-b5b7-b15ca90b5453","Type":"ContainerStarted","Data":"8415044902d64c6371fa3c43a787b2424ce260a38d2ba691d54051946d321a9d"}
Sep 17 00:48:04.056720 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.056601    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:04.071011 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.070965    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6" podStartSLOduration=2.070951857 podStartE2EDuration="2.070951857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:04.06864773 +0000 UTC m=+1080.024244097" watchObservedRunningTime="2026-09-17 00:48:04.070951857 +0000 UTC m=+1080.026548198"
Sep 17 00:48:04.084820 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.084775    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4" podStartSLOduration=1.084760763 podStartE2EDuration="1.084760763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:04.083443451 +0000 UTC m=+1080.039039804" watchObservedRunningTime="2026-09-17 00:48:04.084760763 +0000 UTC m=+1080.040357103"
Sep 17 00:48:04.450623 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.450587    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:04.739021 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.738927    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:04.839404 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.839368    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:04.960009 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:04.959968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:05.053176 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.053114    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:05.187231 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.187199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:05.257351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.257305    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"]
Sep 17 00:48:05.261519 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.261493    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.263945 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.263926    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8fd93c3d-openshift-default-dockercfg-4m6f7\""
Sep 17 00:48:05.272381 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.272360    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"]
Sep 17 00:48:05.273416 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.273329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:05.310397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.310305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f29532-65f2-4508-932d-49c7fd4bd1be-istiod-ca-cert\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.310397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.310351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-data\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.310397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.310381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-certs\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.310656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.310467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-podinfo\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.310656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.310560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-envoy\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.310656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.310628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-token\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.310779 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.310670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-socket\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.310779 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.310711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgp2j\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-kube-api-access-vgp2j\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.310779 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.310737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-credential-socket\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.339199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.339115    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:05.412328 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-envoy\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412497 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-token\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412497 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-socket\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412497 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgp2j\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-kube-api-access-vgp2j\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412667 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-credential-socket\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f29532-65f2-4508-932d-49c7fd4bd1be-istiod-ca-cert\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412787 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-data\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412782    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-socket\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-certs\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412944 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-podinfo\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.412988 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.412964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-credential-socket\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.413082 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.413059    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-certs\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.413173 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.413109    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-data\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.413335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.413313    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f29532-65f2-4508-932d-49c7fd4bd1be-istiod-ca-cert\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.414877 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.414855    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-envoy\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.415595 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.415573    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-podinfo\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.420786 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.420759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-token\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.421358 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.421338    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgp2j\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-kube-api-access-vgp2j\") pod \"e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") " pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.439409 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.439388    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:05.576200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.576091    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:05.861110 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.861083    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"]
Sep 17 00:48:05.863206 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:48:05.863178    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f29532_65f2_4508_932d_49c7fd4bd1be.slice/crio-83ecf71d278fc1e78176daca6bce7fe61c3b04df61e8a32a019bb27937def114 WatchSource:0}: Error finding container 83ecf71d278fc1e78176daca6bce7fe61c3b04df61e8a32a019bb27937def114: Status 404 returned error can't find the container with id 83ecf71d278fc1e78176daca6bce7fe61c3b04df61e8a32a019bb27937def114
Sep 17 00:48:05.866094 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.866057    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:05.866212 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.866158    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:05.866212 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:05.866204    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:06.070453 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:06.070415    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s" event={"ID":"f6f29532-65f2-4508-932d-49c7fd4bd1be","Type":"ContainerStarted","Data":"7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612"}
Sep 17 00:48:06.070663 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:06.070470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s" event={"ID":"f6f29532-65f2-4508-932d-49c7fd4bd1be","Type":"ContainerStarted","Data":"83ecf71d278fc1e78176daca6bce7fe61c3b04df61e8a32a019bb27937def114"}
Sep 17 00:48:06.090135 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:06.090075    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s" podStartSLOduration=1.090060107 podStartE2EDuration="1.090060107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:06.08724795 +0000 UTC m=+1082.042844291" watchObservedRunningTime="2026-09-17 00:48:06.090060107 +0000 UTC m=+1082.045656448"
Sep 17 00:48:06.576963 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:06.576926    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:06.589454 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:06.589418    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:06.757096 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:06.757061    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:07.075249 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:07.075222    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:07.076282 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:07.076259    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:07.556342 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:07.556303    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:07.653841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:07.653806    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:07.762274 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:07.762241    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:07.857095 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:07.857012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:08.552373 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:08.552333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:09.400330 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:09.400296    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:10.099524 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.099385    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:10.199407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.199227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:10.596168 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.587575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:10.698171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.697629    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:10.732688 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.723011    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"]
Sep 17 00:48:10.755173 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.742764    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"]
Sep 17 00:48:10.755173 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.742891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:10.755173 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.746518    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8fd93c3d\""
Sep 17 00:48:10.755173 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.746801    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8fd93c3d-dockercfg-pwbbj\""
Sep 17 00:48:10.765434 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.765405    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:10.787042 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.786829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2pz\" (UniqueName: \"kubernetes.io/projected/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-kube-api-access-2q2pz\") pod \"payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn\" (UID: \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\") " pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:10.787042 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.786887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-plugins-config-volume\") pod \"payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn\" (UID: \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\") " pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:10.892330 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.890798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2pz\" (UniqueName: \"kubernetes.io/projected/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-kube-api-access-2q2pz\") pod \"payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn\" (UID: \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\") " pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:10.892330 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.890862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-plugins-config-volume\") pod \"payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn\" (UID: \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\") " pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:10.892330 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.891904    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-plugins-config-volume\") pod \"payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn\" (UID: \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\") " pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:10.905110 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:10.905034    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2pz\" (UniqueName: \"kubernetes.io/projected/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-kube-api-access-2q2pz\") pod \"payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn\" (UID: \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\") " pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:11.073383 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:11.073341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:11.276176 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:11.275732    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:11.410124 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:11.410072    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:11.426259 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:48:11.426216    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90e7b07_28f5_4901_9dc7_a5c83c3ecc54.slice/crio-2570b9038c584b6b0f0db2f0422a121961e9cfcb013240786a63cfc49ea3e016 WatchSource:0}: Error finding container 2570b9038c584b6b0f0db2f0422a121961e9cfcb013240786a63cfc49ea3e016: Status 404 returned error can't find the container with id 2570b9038c584b6b0f0db2f0422a121961e9cfcb013240786a63cfc49ea3e016
Sep 17 00:48:11.429388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:11.429364    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"]
Sep 17 00:48:11.792616 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:11.792579    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:11.879166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:11.874957    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:12.109083 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:12.108980    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn" event={"ID":"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54","Type":"ContainerStarted","Data":"2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83"}
Sep 17 00:48:12.109083 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:12.109040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn" event={"ID":"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54","Type":"ContainerStarted","Data":"2570b9038c584b6b0f0db2f0422a121961e9cfcb013240786a63cfc49ea3e016"}
Sep 17 00:48:12.109344 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:12.109121    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:12.126017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:12.125807    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn" podStartSLOduration=2.125792047 podStartE2EDuration="2.125792047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:12.123571128 +0000 UTC m=+1088.079167470" watchObservedRunningTime="2026-09-17 00:48:12.125792047 +0000 UTC m=+1088.081388388"
Sep 17 00:48:12.200173 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:12.200062    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:12.359936 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:12.359832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:13.088824 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:13.088777    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:13.925714 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:13.925687    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:14.061162 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:14.060856    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:48:14.061374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:14.061294    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:48:14.390025 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:14.389980    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:14.764907 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:14.764811    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:14.858679 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:14.858648    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:14.958387 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:14.958348    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:15.057751 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:15.057718    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:15.067078 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:15.067053    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:48:15.067361 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:15.067345    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:48:15.551804 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:15.551773    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:15.708440 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:15.708349    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:15.881710 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:15.881662    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:15.962593 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:15.962477    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:16.062365 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:16.062328    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:16.568068 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:16.568026    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:17.443067 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:17.443024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:18.152888 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:18.152837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:18.543208 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:18.543166    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:19.036088 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:19.036047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:19.157237 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:19.157201    2576 generic.go:356] "Generic (PLEG): container finished" podID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerID="f4f655c65eb369b86078520804ac7a787ff1e5c71651d7540f2ed4070c2ef1cd" exitCode=0
Sep 17 00:48:19.157733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:19.157279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" event={"ID":"161c2fcd-59bb-46b4-8899-45cbac6e01ff","Type":"ContainerDied","Data":"f4f655c65eb369b86078520804ac7a787ff1e5c71651d7540f2ed4070c2ef1cd"}
Sep 17 00:48:19.157844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:19.157744    2576 scope.go:117] "RemoveContainer" containerID="f4f655c65eb369b86078520804ac7a787ff1e5c71651d7540f2ed4070c2ef1cd"
Sep 17 00:48:19.159138 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:19.159107    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:19.159244 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:19.159208    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:19.159288 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:19.159243    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:19.757440 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:19.757357    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:19.844071 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:19.843995    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:20.112865 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.112838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:20.168415 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.168375    2576 generic.go:356] "Generic (PLEG): container finished" podID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerID="903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8" exitCode=0
Sep 17 00:48:20.168903 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.168446    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" event={"ID":"1a97aaa1-df48-41b0-a7a3-830e8bbef608","Type":"ContainerDied","Data":"903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8"}
Sep 17 00:48:20.168973 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.168927    2576 scope.go:117] "RemoveContainer" containerID="903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8"
Sep 17 00:48:20.170147 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.170117    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:20.170270 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.170214    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:20.170270 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.170258    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:20.170732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.170714    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" event={"ID":"161c2fcd-59bb-46b4-8899-45cbac6e01ff","Type":"ContainerStarted","Data":"556d812375e3bf8ed0f086d0969f62288b337012adccf01578f8f94974ac1eb6"}
Sep 17 00:48:20.258193 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.257548    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:20.352168 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.351370    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:20.408804 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.408721    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"]
Sep 17 00:48:20.412899 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.412876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.415860 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.415835    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e421ba-openshift-default-dockercfg-blm7r\""
Sep 17 00:48:20.427263 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.426423    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"]
Sep 17 00:48:20.453868 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.452182    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:20.533813 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.533770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:20.567197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.555836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-credential-socket\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.567197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.555907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-envoy\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.567197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.555940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb7738-42b7-4cb1-8b01-65b65835510f-istiod-ca-cert\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.567197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.555985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-data\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.567197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.556061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vtf\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-kube-api-access-f7vtf\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.567197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.556119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-socket\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.567197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.556184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-podinfo\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.567197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.556265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-token\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.567197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.556294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-certs\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.636909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.636824    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.658685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-socket\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.658753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-podinfo\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.658825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-token\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.658855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-certs\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.658905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-credential-socket\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.658936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-envoy\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.658965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb7738-42b7-4cb1-8b01-65b65835510f-istiod-ca-cert\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.658997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-data\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.659061    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vtf\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-kube-api-access-f7vtf\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.659861    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-socket\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.660585    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-credential-socket\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.661076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb7738-42b7-4cb1-8b01-65b65835510f-istiod-ca-cert\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.661583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.661486    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-certs\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.662615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.661621    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-data\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.670011 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.669955    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-podinfo\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.679408 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.678699    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-token\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.679408 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.679027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-envoy\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.679408 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.679366    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vtf\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-kube-api-access-f7vtf\") pod \"e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") " pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.732577 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.732423    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:20.976118 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.976028    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:20.995298 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.995269    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:48:20.995442 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.995387    2576 patch_prober.go:28] interesting pod/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 00:48:20.995442 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:20.995431    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 00:48:21.057169 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:48:21.054712    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cb7738_42b7_4cb1_8b01_65b65835510f.slice/crio-2e389ca63eb4dde8b2c833b82102bc5de3e6569602b5e77e55488114ac28796c WatchSource:0}: Error finding container 2e389ca63eb4dde8b2c833b82102bc5de3e6569602b5e77e55488114ac28796c: Status 404 returned error can't find the container with id 2e389ca63eb4dde8b2c833b82102bc5de3e6569602b5e77e55488114ac28796c
Sep 17 00:48:21.057169 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.055551    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"]
Sep 17 00:48:21.061283 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.058618    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:21.061283 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.058696    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:21.061283 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.058736    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:21.180180 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.179914    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" event={"ID":"1a97aaa1-df48-41b0-a7a3-830e8bbef608","Type":"ContainerStarted","Data":"f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358"}
Sep 17 00:48:21.182911 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.182258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" event={"ID":"e8cb7738-42b7-4cb1-8b01-65b65835510f","Type":"ContainerStarted","Data":"ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184"}
Sep 17 00:48:21.182911 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.182310    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" event={"ID":"e8cb7738-42b7-4cb1-8b01-65b65835510f","Type":"ContainerStarted","Data":"2e389ca63eb4dde8b2c833b82102bc5de3e6569602b5e77e55488114ac28796c"}
Sep 17 00:48:21.218534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.218479    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" podStartSLOduration=1.218462626 podStartE2EDuration="1.218462626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:21.216092354 +0000 UTC m=+1097.171688694" watchObservedRunningTime="2026-09-17 00:48:21.218462626 +0000 UTC m=+1097.174058967"
Sep 17 00:48:21.663504 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.663471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:21.733206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.733177    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:21.995554 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.995464    2576 patch_prober.go:28] interesting pod/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 00:48:21.995554 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:21.995522    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 00:48:22.141930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.141898    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:48:22.142405 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.142372    2576 patch_prober.go:28] interesting pod/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 17 00:48:22.142919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.142835    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 17 00:48:22.626114 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.626078    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"]
Sep 17 00:48:22.636310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.632879    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"
Sep 17 00:48:22.637648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.637526    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"]
Sep 17 00:48:22.641161 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.640434    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:48:22.697438 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.674645    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:22.749603 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.749566    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:22.791174 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.787978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe1197-a577-4c6a-851c-75325420b5a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx\" (UID: \"4dfe1197-a577-4c6a-851c-75325420b5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"
Sep 17 00:48:22.889231 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.888748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe1197-a577-4c6a-851c-75325420b5a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx\" (UID: \"4dfe1197-a577-4c6a-851c-75325420b5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"
Sep 17 00:48:22.893172 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.889632    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe1197-a577-4c6a-851c-75325420b5a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx\" (UID: \"4dfe1197-a577-4c6a-851c-75325420b5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"
Sep 17 00:48:22.966927 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.966881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"
Sep 17 00:48:23.004615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.995924    2576 patch_prober.go:28] interesting pod/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 00:48:23.004615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:22.995989    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 00:48:23.117410 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.116413    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:23.143229 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.142809    2576 patch_prober.go:28] interesting pod/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 17 00:48:23.143229 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.142873    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 17 00:48:23.195079 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.194893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:23.211349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.211279    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:23.224181 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.222486    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:23.295123 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.295049    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:23.664921 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.663533    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"]
Sep 17 00:48:23.995893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.995796    2576 patch_prober.go:28] interesting pod/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 00:48:23.996199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:23.996165    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 00:48:24.080302 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:24.080263    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:24.142892 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:24.142854    2576 patch_prober.go:28] interesting pod/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 17 00:48:24.143087 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:24.142940    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 17 00:48:24.213982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:24.213934    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx" event={"ID":"4dfe1197-a577-4c6a-851c-75325420b5a3","Type":"ContainerStarted","Data":"e1d9196a8f9a095f15ce8f4bc5d64ed7cc88050c68faaf1a5d72f3780a797a47"}
Sep 17 00:48:24.996363 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:24.995860    2576 patch_prober.go:28] interesting pod/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 00:48:24.996363 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:24.995925    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 00:48:25.047040 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:25.047001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:25.143228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:25.142672    2576 patch_prober.go:28] interesting pod/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 17 00:48:25.143228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:25.142738    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 17 00:48:25.223453 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:25.223398    2576 generic.go:356] "Generic (PLEG): container finished" podID="4dfe1197-a577-4c6a-851c-75325420b5a3" containerID="6409055046c734c55502f5ee15a3bd2bce415f81f09257e9bc34dcb053565a61" exitCode=0
Sep 17 00:48:25.223629 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:25.223613    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx" event={"ID":"4dfe1197-a577-4c6a-851c-75325420b5a3","Type":"ContainerDied","Data":"6409055046c734c55502f5ee15a3bd2bce415f81f09257e9bc34dcb053565a61"}
Sep 17 00:48:25.874160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:25.874068    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:25.999820 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:25.999794    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:48:26.000603 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.000577    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:48:26.001298 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.001275    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:48:26.147250 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.144618    2576 patch_prober.go:28] interesting pod/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 17 00:48:26.147250 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.144688    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 17 00:48:26.497635 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.497607    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"
Sep 17 00:48:26.700990 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.700956    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:26.703137 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.703102    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe1197-a577-4c6a-851c-75325420b5a3-maas-api-service-ca\") pod \"4dfe1197-a577-4c6a-851c-75325420b5a3\" (UID: \"4dfe1197-a577-4c6a-851c-75325420b5a3\") "
Sep 17 00:48:26.708738 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.708632    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfe1197-a577-4c6a-851c-75325420b5a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4dfe1197-a577-4c6a-851c-75325420b5a3" (UID: "4dfe1197-a577-4c6a-851c-75325420b5a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:48:26.760229 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.759720    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:26.806884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:26.806842    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfe1197-a577-4c6a-851c-75325420b5a3-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:27.194171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.192692    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:27.194171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.193289    2576 patch_prober.go:28] interesting pod/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 17 00:48:27.194171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.193341    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 17 00:48:27.241181 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.238716    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"
Sep 17 00:48:27.257164 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.250194    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx" event={"ID":"4dfe1197-a577-4c6a-851c-75325420b5a3","Type":"ContainerDied","Data":"e1d9196a8f9a095f15ce8f4bc5d64ed7cc88050c68faaf1a5d72f3780a797a47"}
Sep 17 00:48:27.257164 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.250301    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d9196a8f9a095f15ce8f4bc5d64ed7cc88050c68faaf1a5d72f3780a797a47"
Sep 17 00:48:27.290160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.290075    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:27.304256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.304226    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"]
Sep 17 00:48:27.308746 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.308710    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dfe1197-a577-4c6a-851c-75325420b5a3" containerName="revoke-keys"
Sep 17 00:48:27.308746 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.308742    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dfe1197-a577-4c6a-851c-75325420b5a3" containerName="revoke-keys"
Sep 17 00:48:27.308994 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.308886    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dfe1197-a577-4c6a-851c-75325420b5a3" containerName="revoke-keys"
Sep 17 00:48:27.313753 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.313564    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"]
Sep 17 00:48:27.313753 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.313672    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:27.316271 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.316254    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e421ba-dockercfg-85fhj\""
Sep 17 00:48:27.316720 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.316703    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e421ba\""
Sep 17 00:48:27.415540 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.415373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffca83f-2117-43b0-9846-295c6cda4a5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl\" (UID: \"dffca83f-2117-43b0-9846-295c6cda4a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:27.415540 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.415426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jz84\" (UniqueName: \"kubernetes.io/projected/dffca83f-2117-43b0-9846-295c6cda4a5a-kube-api-access-4jz84\") pod \"payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl\" (UID: \"dffca83f-2117-43b0-9846-295c6cda4a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:27.478172 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.478058    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:27.516199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.516161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffca83f-2117-43b0-9846-295c6cda4a5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl\" (UID: \"dffca83f-2117-43b0-9846-295c6cda4a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:27.516393 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.516251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jz84\" (UniqueName: \"kubernetes.io/projected/dffca83f-2117-43b0-9846-295c6cda4a5a-kube-api-access-4jz84\") pod \"payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl\" (UID: \"dffca83f-2117-43b0-9846-295c6cda4a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:27.516781 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.516759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffca83f-2117-43b0-9846-295c6cda4a5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl\" (UID: \"dffca83f-2117-43b0-9846-295c6cda4a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:27.525551 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.525527    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jz84\" (UniqueName: \"kubernetes.io/projected/dffca83f-2117-43b0-9846-295c6cda4a5a-kube-api-access-4jz84\") pod \"payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl\" (UID: \"dffca83f-2117-43b0-9846-295c6cda4a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:27.586912 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.586879    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:27.626904 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.626868    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:27.835338 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.835308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:27.955264 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:27.955186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"]
Sep 17 00:48:28.151615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:28.151521    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:48:28.152127 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:28.152056    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:48:28.161696 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:28.161671    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:48:28.273311 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:28.273278    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl" event={"ID":"dffca83f-2117-43b0-9846-295c6cda4a5a","Type":"ContainerStarted","Data":"a7dcf206d04d4d76e1ef8ae5dc893bf3cf91a1531d6316db43e491cae6f91373"}
Sep 17 00:48:28.273789 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:28.273340    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:28.273789 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:28.273357    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl" event={"ID":"dffca83f-2117-43b0-9846-295c6cda4a5a","Type":"ContainerStarted","Data":"59ebc3ba0f709fc00074344a40b86ccdc5f5c5f37ea059fbf8c685a4d093c880"}
Sep 17 00:48:28.292772 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:28.292724    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl" podStartSLOduration=1.292708943 podStartE2EDuration="1.292708943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:28.291455336 +0000 UTC m=+1104.247051680" watchObservedRunningTime="2026-09-17 00:48:28.292708943 +0000 UTC m=+1104.248305284"
Sep 17 00:48:28.557010 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:28.556955    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:29.221131 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:29.221100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:29.786338 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:29.786307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:29.954711 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:29.954669    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:30.654388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:30.654350    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:31.270448 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:31.270409    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:32.164279 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.164203    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:32.299216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.299187    2576 generic.go:356] "Generic (PLEG): container finished" podID="f6f29532-65f2-4508-932d-49c7fd4bd1be" containerID="7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612" exitCode=0
Sep 17 00:48:32.299693 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.299268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s" event={"ID":"f6f29532-65f2-4508-932d-49c7fd4bd1be","Type":"ContainerDied","Data":"7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612"}
Sep 17 00:48:32.299782 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.299765    2576 scope.go:117] "RemoveContainer" containerID="7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612"
Sep 17 00:48:32.301200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.301174    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:32.301307 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.301245    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:32.301307 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.301276    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:32.802416 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.802382    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:32.825977 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.825942    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"]
Sep 17 00:48:32.826190 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.826164    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn" podUID="f90e7b07-28f5-4901-9dc7-a5c83c3ecc54" containerName="payload-processing" containerID="cri-o://2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83" gracePeriod=30
Sep 17 00:48:32.895560 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.895523    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:32.976625 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.976585    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"]
Sep 17 00:48:32.980459 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.980420    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:32.981819 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:32.981792    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fd93c3d-d4rqx"]
Sep 17 00:48:33.033286 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.033259    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:33.107582 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.107047    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"]
Sep 17 00:48:33.119962 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.119933    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:33.181266 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.181242    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:33.297424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.297393    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-plugins-config-volume\") pod \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\" (UID: \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\") "
Sep 17 00:48:33.297600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.297452    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q2pz\" (UniqueName: \"kubernetes.io/projected/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-kube-api-access-2q2pz\") pod \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\" (UID: \"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54\") "
Sep 17 00:48:33.297836 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.297807    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f90e7b07-28f5-4901-9dc7-a5c83c3ecc54" (UID: "f90e7b07-28f5-4901-9dc7-a5c83c3ecc54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:48:33.299709 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.299678    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-kube-api-access-2q2pz" (OuterVolumeSpecName: "kube-api-access-2q2pz") pod "f90e7b07-28f5-4901-9dc7-a5c83c3ecc54" (UID: "f90e7b07-28f5-4901-9dc7-a5c83c3ecc54"). InnerVolumeSpecName "kube-api-access-2q2pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:48:33.307974 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.307944    2576 generic.go:356] "Generic (PLEG): container finished" podID="f90e7b07-28f5-4901-9dc7-a5c83c3ecc54" containerID="2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83" exitCode=0
Sep 17 00:48:33.308107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.307983    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"
Sep 17 00:48:33.308107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.308029    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn" event={"ID":"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54","Type":"ContainerDied","Data":"2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83"}
Sep 17 00:48:33.308107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.308092    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn" event={"ID":"f90e7b07-28f5-4901-9dc7-a5c83c3ecc54","Type":"ContainerDied","Data":"2570b9038c584b6b0f0db2f0422a121961e9cfcb013240786a63cfc49ea3e016"}
Sep 17 00:48:33.308309 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.308124    2576 scope.go:117] "RemoveContainer" containerID="2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83"
Sep 17 00:48:33.310576 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.310539    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s" event={"ID":"f6f29532-65f2-4508-932d-49c7fd4bd1be","Type":"ContainerStarted","Data":"1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35"}
Sep 17 00:48:33.310694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.310649    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s" podUID="f6f29532-65f2-4508-932d-49c7fd4bd1be" containerName="istio-proxy" containerID="cri-o://1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35" gracePeriod=30
Sep 17 00:48:33.318018 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.317996    2576 scope.go:117] "RemoveContainer" containerID="2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83"
Sep 17 00:48:33.318321 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:48:33.318302    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83\": container with ID starting with 2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83 not found: ID does not exist" containerID="2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83"
Sep 17 00:48:33.318395 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.318332    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83"} err="failed to get container status \"2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83\": rpc error: code = NotFound desc = could not find container \"2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83\": container with ID starting with 2d08db36bb72bfa89eee466d15d270cb5b4e046cc6a856c12aef6c188d7fea83 not found: ID does not exist"
Sep 17 00:48:33.349168 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.348963    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"]
Sep 17 00:48:33.349168 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.349041    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn"]
Sep 17 00:48:33.399216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.399069    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:33.399216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.399102    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q2pz\" (UniqueName: \"kubernetes.io/projected/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54-kube-api-access-2q2pz\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:33.472374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.472343    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:33.568462 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.568434    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:33.668953 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.668839    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:33.769248 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:33.769214    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:34.107771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.107739    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"]
Sep 17 00:48:34.108355 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.108335    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90e7b07-28f5-4901-9dc7-a5c83c3ecc54" containerName="payload-processing"
Sep 17 00:48:34.108500 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.108485    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90e7b07-28f5-4901-9dc7-a5c83c3ecc54" containerName="payload-processing"
Sep 17 00:48:34.108618 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.108604    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90e7b07-28f5-4901-9dc7-a5c83c3ecc54" containerName="payload-processing"
Sep 17 00:48:34.113317 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.113296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.114892 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.114863    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-8fd93c3d-6c7448599f-b9tzn" podUID="f90e7b07-28f5-4901-9dc7-a5c83c3ecc54" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.48:9004: i/o timeout"
Sep 17 00:48:34.115814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.115797    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ef36871b-gw-openshift-default-dockercfg-v8q4z\""
Sep 17 00:48:34.124640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.124614    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"]
Sep 17 00:48:34.170865 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.163664    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:34.207940 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.207901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-podinfo\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.208275 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.208252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-socket\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.208439 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.208421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdvb\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-kube-api-access-vpdvb\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.208588 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.208569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-envoy\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.208867 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.208841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-data\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.208984 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.208876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-token\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.208984 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.208910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-credential-socket\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.208984 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.208971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-certs\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.209196 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.208996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c448a3-ce75-4490-8363-9ff0e7de69fe-istiod-ca-cert\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.309754 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.309717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-data\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.309754 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.309755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-token\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.309778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-credential-socket\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.309905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-certs\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.309936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c448a3-ce75-4490-8363-9ff0e7de69fe-istiod-ca-cert\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.309972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-podinfo\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.310014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-socket\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.310038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdvb\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-kube-api-access-vpdvb\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.310069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-envoy\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310293 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.310255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-data\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310739 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.310361    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-credential-socket\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310739 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.310376    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-certs\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310739 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.310495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-socket\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.310890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.310774    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c448a3-ce75-4490-8363-9ff0e7de69fe-istiod-ca-cert\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.312924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.312904    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-envoy\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.313094 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.312989    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-podinfo\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.317720 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.317696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-token\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.318094 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.318073    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdvb\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-kube-api-access-vpdvb\") pod \"e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") " pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.390847 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.390755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:34.426190 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.426128    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:34.489237 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.489208    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:34.588061 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.588032    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dfe1197-a577-4c6a-851c-75325420b5a3" path="/var/lib/kubelet/pods/4dfe1197-a577-4c6a-851c-75325420b5a3/volumes"
Sep 17 00:48:34.588442 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.588429    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90e7b07-28f5-4901-9dc7-a5c83c3ecc54" path="/var/lib/kubelet/pods/f90e7b07-28f5-4901-9dc7-a5c83c3ecc54/volumes"
Sep 17 00:48:34.745162 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.736241    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"]
Sep 17 00:48:34.755200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.751308    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:34.755200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.751393    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:34.755200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:34.751433    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:35.181699 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:35.181671    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:35.325186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:35.325128    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" event={"ID":"52c448a3-ce75-4490-8363-9ff0e7de69fe","Type":"ContainerStarted","Data":"7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636"}
Sep 17 00:48:35.325578 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:35.325199    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" event={"ID":"52c448a3-ce75-4490-8363-9ff0e7de69fe","Type":"ContainerStarted","Data":"71b4892846cc763ca96b3c3fd2ed6164715f2e94e7d26a34eb20b0389477b3f2"}
Sep 17 00:48:35.345817 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:35.345743    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" podStartSLOduration=1.345722152 podStartE2EDuration="1.345722152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:35.344438961 +0000 UTC m=+1111.300035304" watchObservedRunningTime="2026-09-17 00:48:35.345722152 +0000 UTC m=+1111.301318494"
Sep 17 00:48:35.427611 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:35.427581    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:35.432692 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:35.432630    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:36.172869 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:36.172835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:36.330243 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:36.330212    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:36.331267 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:36.331250    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:48:36.823529 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:36.823497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:36.879265 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:36.879234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:37.049003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:37.048960    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:37.075860 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:37.075769    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:37.953726 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:37.953652    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:38.876982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:38.876929    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:38.976679 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:38.976641    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:39.151054 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.150947    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:39.283350 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.283316    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:48:39.336592 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.336563    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:39.425615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.425537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:39.473342 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.473312    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"]
Sep 17 00:48:39.478394 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.478370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:39.480731 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.480706    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ef36871b-dockercfg-9wnj5\""
Sep 17 00:48:39.480868 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.480752    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ef36871b\""
Sep 17 00:48:39.488544 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.487073    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"]
Sep 17 00:48:39.574273 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.574238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37990d1-af3e-4879-bb42-5b75be3d2b14-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp\" (UID: \"a37990d1-af3e-4879-bb42-5b75be3d2b14\") " pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:39.574435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.574408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftn47\" (UniqueName: \"kubernetes.io/projected/a37990d1-af3e-4879-bb42-5b75be3d2b14-kube-api-access-ftn47\") pod \"payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp\" (UID: \"a37990d1-af3e-4879-bb42-5b75be3d2b14\") " pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:39.675140 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.675103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftn47\" (UniqueName: \"kubernetes.io/projected/a37990d1-af3e-4879-bb42-5b75be3d2b14-kube-api-access-ftn47\") pod \"payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp\" (UID: \"a37990d1-af3e-4879-bb42-5b75be3d2b14\") " pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:39.675324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.675188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37990d1-af3e-4879-bb42-5b75be3d2b14-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp\" (UID: \"a37990d1-af3e-4879-bb42-5b75be3d2b14\") " pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:39.676087 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.676032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37990d1-af3e-4879-bb42-5b75be3d2b14-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp\" (UID: \"a37990d1-af3e-4879-bb42-5b75be3d2b14\") " pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:39.683106 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.683074    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftn47\" (UniqueName: \"kubernetes.io/projected/a37990d1-af3e-4879-bb42-5b75be3d2b14-kube-api-access-ftn47\") pod \"payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp\" (UID: \"a37990d1-af3e-4879-bb42-5b75be3d2b14\") " pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:39.793186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.793105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:39.963235 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:39.961955    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:40.080533 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:40.080502    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"]
Sep 17 00:48:40.082221 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:48:40.082198    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37990d1_af3e_4879_bb42_5b75be3d2b14.slice/crio-087f668ab88352bee4c4294b4cc2f965a7d732a24458ef0364aadc9804370308 WatchSource:0}: Error finding container 087f668ab88352bee4c4294b4cc2f965a7d732a24458ef0364aadc9804370308: Status 404 returned error can't find the container with id 087f668ab88352bee4c4294b4cc2f965a7d732a24458ef0364aadc9804370308
Sep 17 00:48:40.359901 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:40.358946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp" event={"ID":"a37990d1-af3e-4879-bb42-5b75be3d2b14","Type":"ContainerStarted","Data":"88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266"}
Sep 17 00:48:40.359901 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:40.359016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp" event={"ID":"a37990d1-af3e-4879-bb42-5b75be3d2b14","Type":"ContainerStarted","Data":"087f668ab88352bee4c4294b4cc2f965a7d732a24458ef0364aadc9804370308"}
Sep 17 00:48:40.359901 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:40.359830    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:40.377697 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:40.377633    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp" podStartSLOduration=1.377612709 podStartE2EDuration="1.377612709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:40.374193152 +0000 UTC m=+1116.329789497" watchObservedRunningTime="2026-09-17 00:48:40.377612709 +0000 UTC m=+1116.333209052"
Sep 17 00:48:40.654983 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:40.654885    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:41.453712 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:41.453676    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:42.079450 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:42.079422    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:42.826248 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:42.826217    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:42.923904 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:42.923870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:43.479049 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.479017    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:43.831871 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.831849    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:43.920629 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920596    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-socket\") pod \"f6f29532-65f2-4508-932d-49c7fd4bd1be\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") "
Sep 17 00:48:43.920771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920670    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f29532-65f2-4508-932d-49c7fd4bd1be-istiod-ca-cert\") pod \"f6f29532-65f2-4508-932d-49c7fd4bd1be\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") "
Sep 17 00:48:43.920771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920730    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-podinfo\") pod \"f6f29532-65f2-4508-932d-49c7fd4bd1be\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") "
Sep 17 00:48:43.920771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920756    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgp2j\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-kube-api-access-vgp2j\") pod \"f6f29532-65f2-4508-932d-49c7fd4bd1be\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") "
Sep 17 00:48:43.920934 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920781    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-credential-socket\") pod \"f6f29532-65f2-4508-932d-49c7fd4bd1be\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") "
Sep 17 00:48:43.920934 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920815    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-data\") pod \"f6f29532-65f2-4508-932d-49c7fd4bd1be\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") "
Sep 17 00:48:43.920934 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920875    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-envoy\") pod \"f6f29532-65f2-4508-932d-49c7fd4bd1be\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") "
Sep 17 00:48:43.920934 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920886    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f6f29532-65f2-4508-932d-49c7fd4bd1be" (UID: "f6f29532-65f2-4508-932d-49c7fd4bd1be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:48:43.920934 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920930    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-token\") pod \"f6f29532-65f2-4508-932d-49c7fd4bd1be\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") "
Sep 17 00:48:43.921278 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.920967    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-certs\") pod \"f6f29532-65f2-4508-932d-49c7fd4bd1be\" (UID: \"f6f29532-65f2-4508-932d-49c7fd4bd1be\") "
Sep 17 00:48:43.921278 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.921108    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-data" (OuterVolumeSpecName: "istio-data") pod "f6f29532-65f2-4508-932d-49c7fd4bd1be" (UID: "f6f29532-65f2-4508-932d-49c7fd4bd1be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:48:43.921278 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.921101    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f6f29532-65f2-4508-932d-49c7fd4bd1be" (UID: "f6f29532-65f2-4508-932d-49c7fd4bd1be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:48:43.921278 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.921181    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f29532-65f2-4508-932d-49c7fd4bd1be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f6f29532-65f2-4508-932d-49c7fd4bd1be" (UID: "f6f29532-65f2-4508-932d-49c7fd4bd1be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:48:43.921482 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.921335    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f6f29532-65f2-4508-932d-49c7fd4bd1be" (UID: "f6f29532-65f2-4508-932d-49c7fd4bd1be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:48:43.921482 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.921439    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:43.921482 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.921456    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:43.921482 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.921469    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f29532-65f2-4508-932d-49c7fd4bd1be-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:43.921482 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.921483    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:43.921896 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.921493    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:43.923186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.923161    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f6f29532-65f2-4508-932d-49c7fd4bd1be" (UID: "f6f29532-65f2-4508-932d-49c7fd4bd1be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:48:43.923286 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.923186    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f6f29532-65f2-4508-932d-49c7fd4bd1be" (UID: "f6f29532-65f2-4508-932d-49c7fd4bd1be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:48:43.923674 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.923658    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6f29532-65f2-4508-932d-49c7fd4bd1be" (UID: "f6f29532-65f2-4508-932d-49c7fd4bd1be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:48:43.923723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:43.923654    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-kube-api-access-vgp2j" (OuterVolumeSpecName: "kube-api-access-vgp2j") pod "f6f29532-65f2-4508-932d-49c7fd4bd1be" (UID: "f6f29532-65f2-4508-932d-49c7fd4bd1be"). InnerVolumeSpecName "kube-api-access-vgp2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:48:44.022186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.022120    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:44.022186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.022182    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgp2j\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-kube-api-access-vgp2j\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:44.022186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.022195    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:44.022429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.022204    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f29532-65f2-4508-932d-49c7fd4bd1be-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:48:44.239245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.239126    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:44.387422 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.387386    2576 generic.go:356] "Generic (PLEG): container finished" podID="f6f29532-65f2-4508-932d-49c7fd4bd1be" containerID="1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35" exitCode=0
Sep 17 00:48:44.387592 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.387438    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"
Sep 17 00:48:44.387592 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.387460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s" event={"ID":"f6f29532-65f2-4508-932d-49c7fd4bd1be","Type":"ContainerDied","Data":"1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35"}
Sep 17 00:48:44.387592 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.387504    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s" event={"ID":"f6f29532-65f2-4508-932d-49c7fd4bd1be","Type":"ContainerDied","Data":"83ecf71d278fc1e78176daca6bce7fe61c3b04df61e8a32a019bb27937def114"}
Sep 17 00:48:44.387592 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.387525    2576 scope.go:117] "RemoveContainer" containerID="1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35"
Sep 17 00:48:44.397442 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.397424    2576 scope.go:117] "RemoveContainer" containerID="7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612"
Sep 17 00:48:44.405777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.405759    2576 scope.go:117] "RemoveContainer" containerID="1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35"
Sep 17 00:48:44.406056 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:48:44.406035    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35\": container with ID starting with 1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35 not found: ID does not exist" containerID="1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35"
Sep 17 00:48:44.406111 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.406066    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35"} err="failed to get container status \"1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35\": rpc error: code = NotFound desc = could not find container \"1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35\": container with ID starting with 1233b94022ead1d1d3c652b20ee16a3bfc4a3645232d3834051edf7571f62c35 not found: ID does not exist"
Sep 17 00:48:44.406284 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.406116    2576 scope.go:117] "RemoveContainer" containerID="7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612"
Sep 17 00:48:44.406491 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:48:44.406475    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612\": container with ID starting with 7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612 not found: ID does not exist" containerID="7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612"
Sep 17 00:48:44.406549 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.406496    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612"} err="failed to get container status \"7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612\": rpc error: code = NotFound desc = could not find container \"7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612\": container with ID starting with 7275d72fe59d59408623c22f32eb4c5cc3174ba20da18d3bc7279c5efce2a612 not found: ID does not exist"
Sep 17 00:48:44.407901 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.407881    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"]
Sep 17 00:48:44.413706 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.413676    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8fd93c3d-openshift-default-64487f6f6-rxk4s"]
Sep 17 00:48:44.587733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:44.587697    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6f29532-65f2-4508-932d-49c7fd4bd1be" path="/var/lib/kubelet/pods/f6f29532-65f2-4508-932d-49c7fd4bd1be/volumes"
Sep 17 00:48:45.339266 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:45.339176    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:45.437405 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:45.437368    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:45.546863 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:45.546818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:47.560923 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:47.560860    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:49.419304 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:49.419272    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerID="ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184" exitCode=0
Sep 17 00:48:49.419304 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:49.419315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" event={"ID":"e8cb7738-42b7-4cb1-8b01-65b65835510f","Type":"ContainerDied","Data":"ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184"}
Sep 17 00:48:49.419742 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:49.419624    2576 scope.go:117] "RemoveContainer" containerID="ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184"
Sep 17 00:48:49.420772 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:49.420742    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:49.420878 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:49.420825    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:49.420940 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:49.420874    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:48:50.427186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:50.427128    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" event={"ID":"e8cb7738-42b7-4cb1-8b01-65b65835510f","Type":"ContainerStarted","Data":"1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5"}
Sep 17 00:48:50.733446 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:50.733355    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:50.734521 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:50.734498    2576 patch_prober.go:28] interesting pod/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 00:48:50.734609 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:50.734551    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 00:48:51.447645 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:51.447612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:51.562924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:51.562888    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:51.647775 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:51.647748    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:51.733656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:51.733573    2576 patch_prober.go:28] interesting pod/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 00:48:51.733656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:51.733627    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 00:48:51.783781 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:51.783755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:52.376336 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:52.376304    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:48:52.733350 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:52.733274    2576 patch_prober.go:28] interesting pod/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 00:48:52.733350 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:52.733323    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 00:48:53.204852 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:53.204812    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:53.733678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:53.733642    2576 patch_prober.go:28] interesting pod/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 00:48:53.734173 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:53.733707    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 00:48:54.733605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:54.733555    2576 patch_prober.go:28] interesting pod/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 00:48:54.733804 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:54.733623    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 00:48:55.733258 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:55.733226    2576 patch_prober.go:28] interesting pod/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 00:48:55.733437 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:55.733285    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 00:48:56.741270 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:56.741235    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:56.743222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:56.743192    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:48:57.323096 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:57.323066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:57.419786 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:57.419755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:48:59.860274 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:48:59.860242    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:01.012889 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.012855    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:01.716682 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.716651    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"]
Sep 17 00:49:01.717097 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.717083    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f29532-65f2-4508-932d-49c7fd4bd1be" containerName="istio-proxy"
Sep 17 00:49:01.717161 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.717100    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f29532-65f2-4508-932d-49c7fd4bd1be" containerName="istio-proxy"
Sep 17 00:49:01.717205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.717199    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f29532-65f2-4508-932d-49c7fd4bd1be" containerName="istio-proxy"
Sep 17 00:49:01.717240 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.717205    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f29532-65f2-4508-932d-49c7fd4bd1be" containerName="istio-proxy"
Sep 17 00:49:01.720314 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.720299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"
Sep 17 00:49:01.722774 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.722753    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:49:01.729075 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.728875    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"]
Sep 17 00:49:01.806457 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.806433    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fcea09-7fdd-4f83-9294-9b4410ed01f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx\" (UID: \"b2fcea09-7fdd-4f83-9294-9b4410ed01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"
Sep 17 00:49:01.824494 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.824459    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:01.908079 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.908051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fcea09-7fdd-4f83-9294-9b4410ed01f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx\" (UID: \"b2fcea09-7fdd-4f83-9294-9b4410ed01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"
Sep 17 00:49:01.908647 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:01.908626    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fcea09-7fdd-4f83-9294-9b4410ed01f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx\" (UID: \"b2fcea09-7fdd-4f83-9294-9b4410ed01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"
Sep 17 00:49:02.034064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.034027    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"
Sep 17 00:49:02.309459 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.309437    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"]
Sep 17 00:49:02.311270 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:49:02.311238    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fcea09_7fdd_4f83_9294_9b4410ed01f8.slice/crio-3c8d8ba46fd5302bfe2f137d67e944aee1c3debdc0dcd146944fb73e06141a4d WatchSource:0}: Error finding container 3c8d8ba46fd5302bfe2f137d67e944aee1c3debdc0dcd146944fb73e06141a4d: Status 404 returned error can't find the container with id 3c8d8ba46fd5302bfe2f137d67e944aee1c3debdc0dcd146944fb73e06141a4d
Sep 17 00:49:02.501992 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.501965    2576 generic.go:356] "Generic (PLEG): container finished" podID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerID="7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636" exitCode=0
Sep 17 00:49:02.502122 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.502032    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" event={"ID":"52c448a3-ce75-4490-8363-9ff0e7de69fe","Type":"ContainerDied","Data":"7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636"}
Sep 17 00:49:02.502561 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.502541    2576 scope.go:117] "RemoveContainer" containerID="7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636"
Sep 17 00:49:02.503530 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.503506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx" event={"ID":"b2fcea09-7fdd-4f83-9294-9b4410ed01f8","Type":"ContainerStarted","Data":"3c8d8ba46fd5302bfe2f137d67e944aee1c3debdc0dcd146944fb73e06141a4d"}
Sep 17 00:49:02.503755 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.503728    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:02.503813 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.503801    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:02.503850 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.503834    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:02.545659 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:02.545634    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:03.251284 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:03.251248    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:03.510479 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:03.510385    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" event={"ID":"52c448a3-ce75-4490-8363-9ff0e7de69fe","Type":"ContainerStarted","Data":"b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067"}
Sep 17 00:49:03.511961 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:03.511936    2576 generic.go:356] "Generic (PLEG): container finished" podID="b2fcea09-7fdd-4f83-9294-9b4410ed01f8" containerID="a7f4c07e65ca72fb70fcba41f103d8d5fe575e4019ae3917e0843c4e22fdd0ee" exitCode=0
Sep 17 00:49:03.512079 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:03.511979    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx" event={"ID":"b2fcea09-7fdd-4f83-9294-9b4410ed01f8","Type":"ContainerDied","Data":"a7f4c07e65ca72fb70fcba41f103d8d5fe575e4019ae3917e0843c4e22fdd0ee"}
Sep 17 00:49:03.864791 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:03.864672    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:04.427679 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:04.427636    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:49:04.428982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:04.428953    2576 patch_prober.go:28] interesting pod/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 17 00:49:04.429099 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:04.429002    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 17 00:49:04.468627 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:04.468599    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:04.705454 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:04.705430    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"
Sep 17 00:49:04.765318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:04.755449    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fcea09-7fdd-4f83-9294-9b4410ed01f8-maas-api-service-ca\") pod \"b2fcea09-7fdd-4f83-9294-9b4410ed01f8\" (UID: \"b2fcea09-7fdd-4f83-9294-9b4410ed01f8\") "
Sep 17 00:49:04.765318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:04.756206    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2fcea09-7fdd-4f83-9294-9b4410ed01f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2fcea09-7fdd-4f83-9294-9b4410ed01f8" (UID: "b2fcea09-7fdd-4f83-9294-9b4410ed01f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:49:04.856800 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:04.856757    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fcea09-7fdd-4f83-9294-9b4410ed01f8-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:05.254335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:05.254271    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:05.427346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:05.427310    2576 patch_prober.go:28] interesting pod/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 17 00:49:05.427529 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:05.427377    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 17 00:49:05.523239 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:05.523139    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"
Sep 17 00:49:05.523239 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:05.523179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx" event={"ID":"b2fcea09-7fdd-4f83-9294-9b4410ed01f8","Type":"ContainerDied","Data":"3c8d8ba46fd5302bfe2f137d67e944aee1c3debdc0dcd146944fb73e06141a4d"}
Sep 17 00:49:05.523239 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:05.523227    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8d8ba46fd5302bfe2f137d67e944aee1c3debdc0dcd146944fb73e06141a4d"
Sep 17 00:49:05.863532 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:05.863499    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:06.331124 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:06.331100    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:49:06.427837 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:06.427801    2576 patch_prober.go:28] interesting pod/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 17 00:49:06.428006 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:06.427870    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 17 00:49:06.625578 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:06.625480    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:07.427949 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:07.427903    2576 patch_prober.go:28] interesting pod/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 17 00:49:07.428127 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:07.427957    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 17 00:49:08.067675 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.067644    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:08.427541 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.427454    2576 patch_prober.go:28] interesting pod/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 17 00:49:08.427541 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.427510    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 17 00:49:08.701021 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.700942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:08.724765 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.724730    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"]
Sep 17 00:49:08.725736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.724957    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp" podUID="a37990d1-af3e-4879-bb42-5b75be3d2b14" containerName="payload-processing" containerID="cri-o://88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266" gracePeriod=30
Sep 17 00:49:08.801459 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.801421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:08.878591 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.878553    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:08.878753 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.878690    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"]
Sep 17 00:49:08.883237 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.883201    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ef36871b-8jvmx"]
Sep 17 00:49:08.933934 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:08.933897    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:09.001455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.001428    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"]
Sep 17 00:49:09.001759 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.001671    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy" containerID="cri-o://b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067" gracePeriod=30
Sep 17 00:49:09.014531 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.014512    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:49:09.024389 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.024366    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:09.097882 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.097847    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftn47\" (UniqueName: \"kubernetes.io/projected/a37990d1-af3e-4879-bb42-5b75be3d2b14-kube-api-access-ftn47\") pod \"a37990d1-af3e-4879-bb42-5b75be3d2b14\" (UID: \"a37990d1-af3e-4879-bb42-5b75be3d2b14\") "
Sep 17 00:49:09.098346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.097916    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37990d1-af3e-4879-bb42-5b75be3d2b14-plugins-config-volume\") pod \"a37990d1-af3e-4879-bb42-5b75be3d2b14\" (UID: \"a37990d1-af3e-4879-bb42-5b75be3d2b14\") "
Sep 17 00:49:09.098560 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.098359    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37990d1-af3e-4879-bb42-5b75be3d2b14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a37990d1-af3e-4879-bb42-5b75be3d2b14" (UID: "a37990d1-af3e-4879-bb42-5b75be3d2b14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:49:09.102769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.102739    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37990d1-af3e-4879-bb42-5b75be3d2b14-kube-api-access-ftn47" (OuterVolumeSpecName: "kube-api-access-ftn47") pod "a37990d1-af3e-4879-bb42-5b75be3d2b14" (UID: "a37990d1-af3e-4879-bb42-5b75be3d2b14"). InnerVolumeSpecName "kube-api-access-ftn47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:49:09.198900 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.198859    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftn47\" (UniqueName: \"kubernetes.io/projected/a37990d1-af3e-4879-bb42-5b75be3d2b14-kube-api-access-ftn47\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:09.198900 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.198892    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37990d1-af3e-4879-bb42-5b75be3d2b14-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:09.399962 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.399934    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:09.500615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.500584    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:09.551240 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.551213    2576 generic.go:356] "Generic (PLEG): container finished" podID="a37990d1-af3e-4879-bb42-5b75be3d2b14" containerID="88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266" exitCode=0
Sep 17 00:49:09.551382 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.551267    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"
Sep 17 00:49:09.551382 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.551303    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp" event={"ID":"a37990d1-af3e-4879-bb42-5b75be3d2b14","Type":"ContainerDied","Data":"88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266"}
Sep 17 00:49:09.551382 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.551355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp" event={"ID":"a37990d1-af3e-4879-bb42-5b75be3d2b14","Type":"ContainerDied","Data":"087f668ab88352bee4c4294b4cc2f965a7d732a24458ef0364aadc9804370308"}
Sep 17 00:49:09.551517 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.551385    2576 scope.go:117] "RemoveContainer" containerID="88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266"
Sep 17 00:49:09.561250 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.561229    2576 scope.go:117] "RemoveContainer" containerID="88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266"
Sep 17 00:49:09.561517 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:49:09.561498    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266\": container with ID starting with 88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266 not found: ID does not exist" containerID="88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266"
Sep 17 00:49:09.561588 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.561529    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266"} err="failed to get container status \"88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266\": rpc error: code = NotFound desc = could not find container \"88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266\": container with ID starting with 88bbe63bb499ec0fc0f572d0ed98692cdbafb357446adca3411ab43b2d8d4266 not found: ID does not exist"
Sep 17 00:49:09.573078 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.573056    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"]
Sep 17 00:49:09.583254 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.583224    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ef36871b-5d6f94cd9d-tpppp"]
Sep 17 00:49:09.765622 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.765546    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"]
Sep 17 00:49:09.766272 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.766250    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2fcea09-7fdd-4f83-9294-9b4410ed01f8" containerName="revoke-keys"
Sep 17 00:49:09.766272 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.766274    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2fcea09-7fdd-4f83-9294-9b4410ed01f8" containerName="revoke-keys"
Sep 17 00:49:09.766418 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.766296    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f29532-65f2-4508-932d-49c7fd4bd1be" containerName="istio-proxy"
Sep 17 00:49:09.766418 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.766306    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f29532-65f2-4508-932d-49c7fd4bd1be" containerName="istio-proxy"
Sep 17 00:49:09.766418 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.766319    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37990d1-af3e-4879-bb42-5b75be3d2b14" containerName="payload-processing"
Sep 17 00:49:09.766418 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.766329    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37990d1-af3e-4879-bb42-5b75be3d2b14" containerName="payload-processing"
Sep 17 00:49:09.766544 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.766437    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37990d1-af3e-4879-bb42-5b75be3d2b14" containerName="payload-processing"
Sep 17 00:49:09.766544 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.766449    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2fcea09-7fdd-4f83-9294-9b4410ed01f8" containerName="revoke-keys"
Sep 17 00:49:09.771436 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.771419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:09.774116 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.774097    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1035c6c0-openshift-default-dockercfg-dlkl9\""
Sep 17 00:49:09.780339 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.780314    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"]
Sep 17 00:49:09.789197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.789174    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:09.890200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.890154    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:09.904465 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.904437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-socket\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:09.904583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.904478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-envoy\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:09.904583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.904511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21214503-bcc2-476a-9d26-c51d07d421df-istiod-ca-cert\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:09.904583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.904528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-data\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:09.904583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.904547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2hn\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-kube-api-access-bv2hn\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:09.904743 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.904607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-credential-socket\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:09.904743 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.904659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-istio-token\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:09.904743 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.904686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-certs\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:09.904743 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:09.904708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21214503-bcc2-476a-9d26-c51d07d421df-istio-podinfo\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.005648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.005616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21214503-bcc2-476a-9d26-c51d07d421df-istio-podinfo\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.005858 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.005666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-socket\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.005858 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.005696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-envoy\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.005858 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.005737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21214503-bcc2-476a-9d26-c51d07d421df-istiod-ca-cert\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.005858 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.005756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-data\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.005858 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.005785    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2hn\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-kube-api-access-bv2hn\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.005858 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.005823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-credential-socket\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.006233 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.005863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-istio-token\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.006233 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.005901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-certs\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.006233 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.006113    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-socket\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.006233 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.006183    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-data\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.006439 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.006416    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-credential-socket\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.006499 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.006437    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-certs\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.006553 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.006541    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21214503-bcc2-476a-9d26-c51d07d421df-istiod-ca-cert\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.008308 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.008289    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-envoy\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.008445 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.008430    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21214503-bcc2-476a-9d26-c51d07d421df-istio-podinfo\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.012980 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.012959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-istio-token\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.013179 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.013136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2hn\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-kube-api-access-bv2hn\") pod \"e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") " pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.084128 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.084106    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:10.152746 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.150719    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:10.366061 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.366040    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"]
Sep 17 00:49:10.367960 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:49:10.367935    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21214503_bcc2_476a_9d26_c51d07d421df.slice/crio-cfc2bafa38c2001e91ff6fc14084da615c1cd5444ad10189e1af77ca5dc46d65 WatchSource:0}: Error finding container cfc2bafa38c2001e91ff6fc14084da615c1cd5444ad10189e1af77ca5dc46d65: Status 404 returned error can't find the container with id cfc2bafa38c2001e91ff6fc14084da615c1cd5444ad10189e1af77ca5dc46d65
Sep 17 00:49:10.370896 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.370862    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:10.370996 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.370926    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:10.370996 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.370957    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:10.560399 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.560356    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" event={"ID":"21214503-bcc2-476a-9d26-c51d07d421df","Type":"ContainerStarted","Data":"66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0"}
Sep 17 00:49:10.560591 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.560426    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" event={"ID":"21214503-bcc2-476a-9d26-c51d07d421df","Type":"ContainerStarted","Data":"cfc2bafa38c2001e91ff6fc14084da615c1cd5444ad10189e1af77ca5dc46d65"}
Sep 17 00:49:10.578080 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.578029    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" podStartSLOduration=1.578013705 podStartE2EDuration="1.578013705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:49:10.577355675 +0000 UTC m=+1146.532952018" watchObservedRunningTime="2026-09-17 00:49:10.578013705 +0000 UTC m=+1146.533610046"
Sep 17 00:49:10.588757 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.588731    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37990d1-af3e-4879-bb42-5b75be3d2b14" path="/var/lib/kubelet/pods/a37990d1-af3e-4879-bb42-5b75be3d2b14/volumes"
Sep 17 00:49:10.589092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:10.589078    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2fcea09-7fdd-4f83-9294-9b4410ed01f8" path="/var/lib/kubelet/pods/b2fcea09-7fdd-4f83-9294-9b4410ed01f8/volumes"
Sep 17 00:49:11.044748 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:11.044716    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:11.084524 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:11.084496    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:11.089333 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:11.089309    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:11.566624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:11.566589    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:11.567461 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:11.567442    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:11.764512 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:11.764471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:12.056089 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:12.056049    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:12.153216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:12.153181    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:12.590676 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:12.590637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:13.394487 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:13.394454    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:14.012813 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.012787    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:14.111810 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.111783    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:14.255612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.255587    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:49:14.349216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349177    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-certs\") pod \"52c448a3-ce75-4490-8363-9ff0e7de69fe\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") "
Sep 17 00:49:14.349216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349219    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-data\") pod \"52c448a3-ce75-4490-8363-9ff0e7de69fe\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") "
Sep 17 00:49:14.349466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349280    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpdvb\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-kube-api-access-vpdvb\") pod \"52c448a3-ce75-4490-8363-9ff0e7de69fe\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") "
Sep 17 00:49:14.349466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349296    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-token\") pod \"52c448a3-ce75-4490-8363-9ff0e7de69fe\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") "
Sep 17 00:49:14.349466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349320    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c448a3-ce75-4490-8363-9ff0e7de69fe-istiod-ca-cert\") pod \"52c448a3-ce75-4490-8363-9ff0e7de69fe\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") "
Sep 17 00:49:14.349466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349383    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52c448a3-ce75-4490-8363-9ff0e7de69fe" (UID: "52c448a3-ce75-4490-8363-9ff0e7de69fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:14.349466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349390    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-podinfo\") pod \"52c448a3-ce75-4490-8363-9ff0e7de69fe\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") "
Sep 17 00:49:14.349466 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-credential-socket\") pod \"52c448a3-ce75-4490-8363-9ff0e7de69fe\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") "
Sep 17 00:49:14.349782 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349473    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "52c448a3-ce75-4490-8363-9ff0e7de69fe" (UID: "52c448a3-ce75-4490-8363-9ff0e7de69fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:14.349782 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349502    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-envoy\") pod \"52c448a3-ce75-4490-8363-9ff0e7de69fe\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") "
Sep 17 00:49:14.349782 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349529    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-socket\") pod \"52c448a3-ce75-4490-8363-9ff0e7de69fe\" (UID: \"52c448a3-ce75-4490-8363-9ff0e7de69fe\") "
Sep 17 00:49:14.349930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349843    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52c448a3-ce75-4490-8363-9ff0e7de69fe" (UID: "52c448a3-ce75-4490-8363-9ff0e7de69fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:14.349930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349860    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c448a3-ce75-4490-8363-9ff0e7de69fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52c448a3-ce75-4490-8363-9ff0e7de69fe" (UID: "52c448a3-ce75-4490-8363-9ff0e7de69fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:49:14.350032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349960    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52c448a3-ce75-4490-8363-9ff0e7de69fe" (UID: "52c448a3-ce75-4490-8363-9ff0e7de69fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:14.350032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.349988    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:14.350032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.350008    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:14.350032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.350022    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:14.350241 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.350034    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c448a3-ce75-4490-8363-9ff0e7de69fe-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:14.351689 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.351657    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "52c448a3-ce75-4490-8363-9ff0e7de69fe" (UID: "52c448a3-ce75-4490-8363-9ff0e7de69fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:49:14.351806 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.351706    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52c448a3-ce75-4490-8363-9ff0e7de69fe" (UID: "52c448a3-ce75-4490-8363-9ff0e7de69fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:14.352087 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.352067    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-kube-api-access-vpdvb" (OuterVolumeSpecName: "kube-api-access-vpdvb") pod "52c448a3-ce75-4490-8363-9ff0e7de69fe" (UID: "52c448a3-ce75-4490-8363-9ff0e7de69fe"). InnerVolumeSpecName "kube-api-access-vpdvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:49:14.352161 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.352123    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52c448a3-ce75-4490-8363-9ff0e7de69fe" (UID: "52c448a3-ce75-4490-8363-9ff0e7de69fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:49:14.356523 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.356495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:14.450750 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.450718    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpdvb\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-kube-api-access-vpdvb\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:14.450750 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.450745    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:14.450750 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.450757    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:14.450943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.450770    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:14.450943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.450781    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c448a3-ce75-4490-8363-9ff0e7de69fe-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:14.470433 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.470389    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:14.568295 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.568260    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:14.584119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.584091    2576 generic.go:356] "Generic (PLEG): container finished" podID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerID="b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067" exitCode=0
Sep 17 00:49:14.584248 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.584215    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"
Sep 17 00:49:14.587757 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.587733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" event={"ID":"52c448a3-ce75-4490-8363-9ff0e7de69fe","Type":"ContainerDied","Data":"b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067"}
Sep 17 00:49:14.587863 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.587765    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp" event={"ID":"52c448a3-ce75-4490-8363-9ff0e7de69fe","Type":"ContainerDied","Data":"71b4892846cc763ca96b3c3fd2ed6164715f2e94e7d26a34eb20b0389477b3f2"}
Sep 17 00:49:14.587863 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.587788    2576 scope.go:117] "RemoveContainer" containerID="b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067"
Sep 17 00:49:14.597066 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.597048    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"]
Sep 17 00:49:14.597489 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.597474    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy"
Sep 17 00:49:14.597545 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.597490    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy"
Sep 17 00:49:14.597582 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.597552    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy"
Sep 17 00:49:14.597582 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.597559    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy"
Sep 17 00:49:14.598407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.598387    2576 scope.go:117] "RemoveContainer" containerID="7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636"
Sep 17 00:49:14.601847 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.601833    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:14.603969 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.603948    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1035c6c0-dockercfg-k6vrz\""
Sep 17 00:49:14.604231 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.604212    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1035c6c0\""
Sep 17 00:49:14.606908 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.606892    2576 scope.go:117] "RemoveContainer" containerID="b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067"
Sep 17 00:49:14.607259 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:49:14.607239    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067\": container with ID starting with b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067 not found: ID does not exist" containerID="b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067"
Sep 17 00:49:14.607333 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.607272    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067"} err="failed to get container status \"b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067\": rpc error: code = NotFound desc = could not find container \"b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067\": container with ID starting with b3f734b9380e336967df37c9cf331ff47b320674e4cb21c403b6c26945745067 not found: ID does not exist"
Sep 17 00:49:14.607333 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.607296    2576 scope.go:117] "RemoveContainer" containerID="7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636"
Sep 17 00:49:14.607624 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:49:14.607596    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636\": container with ID starting with 7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636 not found: ID does not exist" containerID="7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636"
Sep 17 00:49:14.607737 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.607630    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636"} err="failed to get container status \"7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636\": rpc error: code = NotFound desc = could not find container \"7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636\": container with ID starting with 7fd408a696bc6f7b71f717de0b387adbefab3eab5cd8b80ad1e6cbb14deef636 not found: ID does not exist"
Sep 17 00:49:14.611534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.611452    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"]
Sep 17 00:49:14.614676 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.614634    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"]
Sep 17 00:49:14.618516 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.618495    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ef36871b-gw-openshift-default-57f59448dd-rkktp"]
Sep 17 00:49:14.753436 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.753402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a09b69a-2b12-4508-bdaf-d346529522ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb\" (UID: \"3a09b69a-2b12-4508-bdaf-d346529522ed\") " pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:14.753597 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.753450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjd7b\" (UniqueName: \"kubernetes.io/projected/3a09b69a-2b12-4508-bdaf-d346529522ed-kube-api-access-gjd7b\") pod \"payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb\" (UID: \"3a09b69a-2b12-4508-bdaf-d346529522ed\") " pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:14.854479 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.854402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a09b69a-2b12-4508-bdaf-d346529522ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb\" (UID: \"3a09b69a-2b12-4508-bdaf-d346529522ed\") " pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:14.854479 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.854445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjd7b\" (UniqueName: \"kubernetes.io/projected/3a09b69a-2b12-4508-bdaf-d346529522ed-kube-api-access-gjd7b\") pod \"payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb\" (UID: \"3a09b69a-2b12-4508-bdaf-d346529522ed\") " pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:14.855045 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.855027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a09b69a-2b12-4508-bdaf-d346529522ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb\" (UID: \"3a09b69a-2b12-4508-bdaf-d346529522ed\") " pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:14.862151 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.862130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjd7b\" (UniqueName: \"kubernetes.io/projected/3a09b69a-2b12-4508-bdaf-d346529522ed-kube-api-access-gjd7b\") pod \"payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb\" (UID: \"3a09b69a-2b12-4508-bdaf-d346529522ed\") " pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:14.869085 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.869059    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:14.913070 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:14.913046    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:15.194055 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:15.194032    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"]
Sep 17 00:49:15.196108 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:49:15.196081    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a09b69a_2b12_4508_bdaf_d346529522ed.slice/crio-c26e9d9531e757472d22f44e25c61f545b04d374444c97407e8a99b04527ae81 WatchSource:0}: Error finding container c26e9d9531e757472d22f44e25c61f545b04d374444c97407e8a99b04527ae81: Status 404 returned error can't find the container with id c26e9d9531e757472d22f44e25c61f545b04d374444c97407e8a99b04527ae81
Sep 17 00:49:15.544217 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:15.544185    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:15.592568 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:15.592540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb" event={"ID":"3a09b69a-2b12-4508-bdaf-d346529522ed","Type":"ContainerStarted","Data":"7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a"}
Sep 17 00:49:15.592715 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:15.592586    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb" event={"ID":"3a09b69a-2b12-4508-bdaf-d346529522ed","Type":"ContainerStarted","Data":"c26e9d9531e757472d22f44e25c61f545b04d374444c97407e8a99b04527ae81"}
Sep 17 00:49:15.592715 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:15.592641    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:15.610453 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:15.610408    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb" podStartSLOduration=1.6103947650000001 podStartE2EDuration="1.610394765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:49:15.609266248 +0000 UTC m=+1151.564862590" watchObservedRunningTime="2026-09-17 00:49:15.610394765 +0000 UTC m=+1151.565991105"
Sep 17 00:49:16.268320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:16.268283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:16.588689 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:16.588654    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" path="/var/lib/kubelet/pods/52c448a3-ce75-4490-8363-9ff0e7de69fe/volumes"
Sep 17 00:49:16.843184 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:16.843090    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:26.600115 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:26.600035    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:27.270495 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:27.270463    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:34.707987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:34.707909    2576 generic.go:356] "Generic (PLEG): container finished" podID="21214503-bcc2-476a-9d26-c51d07d421df" containerID="66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0" exitCode=0
Sep 17 00:49:34.707987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:34.707968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" event={"ID":"21214503-bcc2-476a-9d26-c51d07d421df","Type":"ContainerDied","Data":"66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0"}
Sep 17 00:49:34.708439 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:34.708423    2576 scope.go:117] "RemoveContainer" containerID="66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0"
Sep 17 00:49:34.709705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:34.709675    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:34.709796 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:34.709745    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:34.709796 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:34.709776    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:34.863771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:34.863740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:35.715292 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:35.715257    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" event={"ID":"21214503-bcc2-476a-9d26-c51d07d421df","Type":"ContainerStarted","Data":"d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621"}
Sep 17 00:49:36.084510 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:36.084471    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:36.084680 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:36.084640    2576 patch_prober.go:28] interesting pod/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 17 00:49:36.084727 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:36.084696    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 00:49:36.104528 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:36.104501    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:36.854834 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:36.854803    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:37.084720 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.084689    2576 patch_prober.go:28] interesting pod/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 17 00:49:37.084883 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.084744    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 00:49:37.475317 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.475286    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:37.707237 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.707204    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"]
Sep 17 00:49:37.707642 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.707630    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy"
Sep 17 00:49:37.707700 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.707644    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c448a3-ce75-4490-8363-9ff0e7de69fe" containerName="istio-proxy"
Sep 17 00:49:37.710996 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.710979    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"
Sep 17 00:49:37.713369 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.713350    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:49:37.718494 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.718244    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"]
Sep 17 00:49:37.778747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.778714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7302958b-cf15-4a2a-a59c-79140c2b4d80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b\" (UID: \"7302958b-cf15-4a2a-a59c-79140c2b4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"
Sep 17 00:49:37.880180 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.880132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7302958b-cf15-4a2a-a59c-79140c2b4d80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b\" (UID: \"7302958b-cf15-4a2a-a59c-79140c2b4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"
Sep 17 00:49:37.880878 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:37.880852    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7302958b-cf15-4a2a-a59c-79140c2b4d80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b\" (UID: \"7302958b-cf15-4a2a-a59c-79140c2b4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"
Sep 17 00:49:38.022924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:38.022895    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"
Sep 17 00:49:38.084625 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:38.084581    2576 patch_prober.go:28] interesting pod/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 17 00:49:38.084802 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:38.084648    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 00:49:38.247805 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:38.247708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:38.326864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:38.326800    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"]
Sep 17 00:49:38.327372 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:49:38.327298    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7302958b_cf15_4a2a_a59c_79140c2b4d80.slice/crio-8cfaa4ee2c8dfb17b514b19a909c4c61deb53f657c515960dd532afd4afb95a2 WatchSource:0}: Error finding container 8cfaa4ee2c8dfb17b514b19a909c4c61deb53f657c515960dd532afd4afb95a2: Status 404 returned error can't find the container with id 8cfaa4ee2c8dfb17b514b19a909c4c61deb53f657c515960dd532afd4afb95a2
Sep 17 00:49:38.734620 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:38.734594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b" event={"ID":"7302958b-cf15-4a2a-a59c-79140c2b4d80","Type":"ContainerStarted","Data":"8cfaa4ee2c8dfb17b514b19a909c4c61deb53f657c515960dd532afd4afb95a2"}
Sep 17 00:49:39.035647 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:39.035612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:39.084769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:39.084742    2576 patch_prober.go:28] interesting pod/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 17 00:49:39.084924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:39.084794    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 00:49:39.742610 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:39.742482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:39.745536 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:39.745495    2576 generic.go:356] "Generic (PLEG): container finished" podID="7302958b-cf15-4a2a-a59c-79140c2b4d80" containerID="068b66ccc11c78805ec0e35023d710466115de59df20c6187cd714fd627c294f" exitCode=0
Sep 17 00:49:39.745806 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:39.745781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b" event={"ID":"7302958b-cf15-4a2a-a59c-79140c2b4d80","Type":"ContainerDied","Data":"068b66ccc11c78805ec0e35023d710466115de59df20c6187cd714fd627c294f"}
Sep 17 00:49:40.084605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:40.084559    2576 patch_prober.go:28] interesting pod/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 17 00:49:40.084980 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:40.084628    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 00:49:40.365993 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:40.365867    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:40.893704 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:40.893679    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"
Sep 17 00:49:41.010413 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.010349    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7302958b-cf15-4a2a-a59c-79140c2b4d80-maas-api-service-ca\") pod \"7302958b-cf15-4a2a-a59c-79140c2b4d80\" (UID: \"7302958b-cf15-4a2a-a59c-79140c2b4d80\") "
Sep 17 00:49:41.010708 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.010682    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7302958b-cf15-4a2a-a59c-79140c2b4d80-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7302958b-cf15-4a2a-a59c-79140c2b4d80" (UID: "7302958b-cf15-4a2a-a59c-79140c2b4d80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:49:41.059446 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.059421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:41.088758 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.088735    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:41.089100 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.088997    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:41.089684 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.089670    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:41.113241 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.112050    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7302958b-cf15-4a2a-a59c-79140c2b4d80-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:41.662400 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.662368    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:41.760007 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.759980    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"
Sep 17 00:49:41.760206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.760016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b" event={"ID":"7302958b-cf15-4a2a-a59c-79140c2b4d80","Type":"ContainerDied","Data":"8cfaa4ee2c8dfb17b514b19a909c4c61deb53f657c515960dd532afd4afb95a2"}
Sep 17 00:49:41.760206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:41.760066    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cfaa4ee2c8dfb17b514b19a909c4c61deb53f657c515960dd532afd4afb95a2"
Sep 17 00:49:42.309516 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:42.309488    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:43.380247 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:43.380208    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:44.025208 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.025177    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:44.048634 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.048602    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"]
Sep 17 00:49:44.048857 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.048831    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb" podUID="3a09b69a-2b12-4508-bdaf-d346529522ed" containerName="payload-processing" containerID="cri-o://7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a" gracePeriod=30
Sep 17 00:49:44.133306 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.133264    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:44.262890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.262521    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:44.283931 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.283881    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"]
Sep 17 00:49:44.292193 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.292163    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1035c6c0-2jd5b"]
Sep 17 00:49:44.315860 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.315826    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:44.334435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.334413    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:44.448481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.448450    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjd7b\" (UniqueName: \"kubernetes.io/projected/3a09b69a-2b12-4508-bdaf-d346529522ed-kube-api-access-gjd7b\") pod \"3a09b69a-2b12-4508-bdaf-d346529522ed\" (UID: \"3a09b69a-2b12-4508-bdaf-d346529522ed\") "
Sep 17 00:49:44.448931 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.448495    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a09b69a-2b12-4508-bdaf-d346529522ed-plugins-config-volume\") pod \"3a09b69a-2b12-4508-bdaf-d346529522ed\" (UID: \"3a09b69a-2b12-4508-bdaf-d346529522ed\") "
Sep 17 00:49:44.448931 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.448823    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a09b69a-2b12-4508-bdaf-d346529522ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a09b69a-2b12-4508-bdaf-d346529522ed" (UID: "3a09b69a-2b12-4508-bdaf-d346529522ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:49:44.450730 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.450706    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a09b69a-2b12-4508-bdaf-d346529522ed-kube-api-access-gjd7b" (OuterVolumeSpecName: "kube-api-access-gjd7b") pod "3a09b69a-2b12-4508-bdaf-d346529522ed" (UID: "3a09b69a-2b12-4508-bdaf-d346529522ed"). InnerVolumeSpecName "kube-api-access-gjd7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:49:44.550266 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.550201    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a09b69a-2b12-4508-bdaf-d346529522ed-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:44.550266 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.550229    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjd7b\" (UniqueName: \"kubernetes.io/projected/3a09b69a-2b12-4508-bdaf-d346529522ed-kube-api-access-gjd7b\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:44.560535 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.560514    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:44.588157 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.588123    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7302958b-cf15-4a2a-a59c-79140c2b4d80" path="/var/lib/kubelet/pods/7302958b-cf15-4a2a-a59c-79140c2b4d80/volumes"
Sep 17 00:49:44.776472 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.776441    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a09b69a-2b12-4508-bdaf-d346529522ed" containerID="7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a" exitCode=0
Sep 17 00:49:44.776629 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.776485    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"
Sep 17 00:49:44.776629 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.776520    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb" event={"ID":"3a09b69a-2b12-4508-bdaf-d346529522ed","Type":"ContainerDied","Data":"7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a"}
Sep 17 00:49:44.776629 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.776573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb" event={"ID":"3a09b69a-2b12-4508-bdaf-d346529522ed","Type":"ContainerDied","Data":"c26e9d9531e757472d22f44e25c61f545b04d374444c97407e8a99b04527ae81"}
Sep 17 00:49:44.776629 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.776604    2576 scope.go:117] "RemoveContainer" containerID="7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a"
Sep 17 00:49:44.785223 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.785204    2576 scope.go:117] "RemoveContainer" containerID="7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a"
Sep 17 00:49:44.785488 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:49:44.785470    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a\": container with ID starting with 7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a not found: ID does not exist" containerID="7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a"
Sep 17 00:49:44.785549 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.785493    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a"} err="failed to get container status \"7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a\": rpc error: code = NotFound desc = could not find container \"7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a\": container with ID starting with 7c335395cf16dd33146a6c31b00b0ffdab27bce1e48edf802a7345efffc13c2a not found: ID does not exist"
Sep 17 00:49:44.791653 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.791632    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"]
Sep 17 00:49:44.795544 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:44.795522    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1035c6c0-5497b8dc69-6fgbb"]
Sep 17 00:49:45.331379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:45.331343    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:45.371887 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:45.371856    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"]
Sep 17 00:49:45.372100 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:45.372076    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy" containerID="cri-o://d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621" gracePeriod=30
Sep 17 00:49:45.400182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:45.400135    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:45.811294 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:45.811255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:45.959510 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:45.959481    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:46.080917 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.080835    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"]
Sep 17 00:49:46.081496 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.081476    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a09b69a-2b12-4508-bdaf-d346529522ed" containerName="payload-processing"
Sep 17 00:49:46.081496 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.081496    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a09b69a-2b12-4508-bdaf-d346529522ed" containerName="payload-processing"
Sep 17 00:49:46.081633 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.081523    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7302958b-cf15-4a2a-a59c-79140c2b4d80" containerName="revoke-keys"
Sep 17 00:49:46.081633 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.081531    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7302958b-cf15-4a2a-a59c-79140c2b4d80" containerName="revoke-keys"
Sep 17 00:49:46.081695 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.081658    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7302958b-cf15-4a2a-a59c-79140c2b4d80" containerName="revoke-keys"
Sep 17 00:49:46.081695 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.081676    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a09b69a-2b12-4508-bdaf-d346529522ed" containerName="payload-processing"
Sep 17 00:49:46.086359 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.086332    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.089528 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.089511    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-39a5d066-openshift-default-dockercfg-vm7bx\""
Sep 17 00:49:46.093687 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.093667    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"]
Sep 17 00:49:46.111329 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.111300    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:46.165636 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.165613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-certs\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.165726 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.165648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-podinfo\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.165726 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.165667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-token\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.165823 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.165745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-socket\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.165823 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.165776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-envoy\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.165898 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.165875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-credential-socket\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.165956 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.165902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab41c5-983b-41ee-addb-43fbead3b0cf-istiod-ca-cert\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.166011 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.165955    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-data\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.166064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.166005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2q55\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-kube-api-access-g2q55\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.266634 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.266600    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:46.266778 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.266756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-podinfo\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.266849 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.266786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-token\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.266849 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.266837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-socket\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.266943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.266866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-envoy\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.266999 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.266982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-credential-socket\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.267085 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.267007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab41c5-983b-41ee-addb-43fbead3b0cf-istiod-ca-cert\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.267085 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.267045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-data\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.267226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.267089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2q55\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-kube-api-access-g2q55\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.267226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.267127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-certs\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.267345 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.267323    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-socket\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.267665 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.267568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-credential-socket\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.267845 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.267747    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-certs\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.267845 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.267782    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-data\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.267962 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.267895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab41c5-983b-41ee-addb-43fbead3b0cf-istiod-ca-cert\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.269454 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.269434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-envoy\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.269576 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.269493    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-podinfo\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.274188 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.274137    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-token\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.274300 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.274280    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2q55\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-kube-api-access-g2q55\") pod \"e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") " pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.399368 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.399294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:46.588000 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.587973    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a09b69a-2b12-4508-bdaf-d346529522ed" path="/var/lib/kubelet/pods/3a09b69a-2b12-4508-bdaf-d346529522ed/volumes"
Sep 17 00:49:46.688493 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.688395    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"]
Sep 17 00:49:46.690710 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:49:46.690678    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eab41c5_983b_41ee_addb_43fbead3b0cf.slice/crio-7dbf902646f6a6bec74dc6e80ce5c837e29714402567c070d88b8c11f5c0e21c WatchSource:0}: Error finding container 7dbf902646f6a6bec74dc6e80ce5c837e29714402567c070d88b8c11f5c0e21c: Status 404 returned error can't find the container with id 7dbf902646f6a6bec74dc6e80ce5c837e29714402567c070d88b8c11f5c0e21c
Sep 17 00:49:46.692784 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.692751    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:46.692856 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.692815    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:46.692856 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.692843    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:49:46.718804 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.718784    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:46.793318 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.793279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" event={"ID":"2eab41c5-983b-41ee-addb-43fbead3b0cf","Type":"ContainerStarted","Data":"e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c"}
Sep 17 00:49:46.793507 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.793345    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" event={"ID":"2eab41c5-983b-41ee-addb-43fbead3b0cf","Type":"ContainerStarted","Data":"7dbf902646f6a6bec74dc6e80ce5c837e29714402567c070d88b8c11f5c0e21c"}
Sep 17 00:49:46.810639 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:46.810435    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" podStartSLOduration=0.810417584 podStartE2EDuration="810.417584ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:49:46.809205163 +0000 UTC m=+1182.764801504" watchObservedRunningTime="2026-09-17 00:49:46.810417584 +0000 UTC m=+1182.766013927"
Sep 17 00:49:47.365067 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:47.365034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:47.399475 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:47.399439    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:47.404802 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:47.404776    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:47.798883 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:47.798859    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:47.799877 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:47.799859    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:49:48.079081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:48.079001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:48.174502 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:48.174475    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:48.277653 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:48.277625    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:48.946065 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:48.946034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:49.581036 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:49.580991    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:50.222834 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.222802    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:50.629678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.629657    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:50.709809 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.709783    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv2hn\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-kube-api-access-bv2hn\") pod \"21214503-bcc2-476a-9d26-c51d07d421df\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") "
Sep 17 00:49:50.709931 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.709819    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-certs\") pod \"21214503-bcc2-476a-9d26-c51d07d421df\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") "
Sep 17 00:49:50.709931 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.709858    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-istio-token\") pod \"21214503-bcc2-476a-9d26-c51d07d421df\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") "
Sep 17 00:49:50.709931 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.709878    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-data\") pod \"21214503-bcc2-476a-9d26-c51d07d421df\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") "
Sep 17 00:49:50.709931 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.709913    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-envoy\") pod \"21214503-bcc2-476a-9d26-c51d07d421df\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") "
Sep 17 00:49:50.710102 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710000    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-credential-socket\") pod \"21214503-bcc2-476a-9d26-c51d07d421df\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") "
Sep 17 00:49:50.710102 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710041    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21214503-bcc2-476a-9d26-c51d07d421df-istio-podinfo\") pod \"21214503-bcc2-476a-9d26-c51d07d421df\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") "
Sep 17 00:49:50.710102 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710072    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-socket\") pod \"21214503-bcc2-476a-9d26-c51d07d421df\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") "
Sep 17 00:49:50.710277 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710108    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21214503-bcc2-476a-9d26-c51d07d421df-istiod-ca-cert\") pod \"21214503-bcc2-476a-9d26-c51d07d421df\" (UID: \"21214503-bcc2-476a-9d26-c51d07d421df\") "
Sep 17 00:49:50.710277 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710221    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21214503-bcc2-476a-9d26-c51d07d421df" (UID: "21214503-bcc2-476a-9d26-c51d07d421df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:50.710277 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710245    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-data" (OuterVolumeSpecName: "istio-data") pod "21214503-bcc2-476a-9d26-c51d07d421df" (UID: "21214503-bcc2-476a-9d26-c51d07d421df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:50.710520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710483    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21214503-bcc2-476a-9d26-c51d07d421df" (UID: "21214503-bcc2-476a-9d26-c51d07d421df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:50.710520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710496    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21214503-bcc2-476a-9d26-c51d07d421df" (UID: "21214503-bcc2-476a-9d26-c51d07d421df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:50.710681 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710584    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21214503-bcc2-476a-9d26-c51d07d421df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21214503-bcc2-476a-9d26-c51d07d421df" (UID: "21214503-bcc2-476a-9d26-c51d07d421df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:49:50.710976 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710951    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:50.711095 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710982    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21214503-bcc2-476a-9d26-c51d07d421df-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:50.711095 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.710997    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:50.711095 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.711009    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:50.711095 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.711022    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:50.712320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.712292    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-kube-api-access-bv2hn" (OuterVolumeSpecName: "kube-api-access-bv2hn") pod "21214503-bcc2-476a-9d26-c51d07d421df" (UID: "21214503-bcc2-476a-9d26-c51d07d421df"). InnerVolumeSpecName "kube-api-access-bv2hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:49:50.712320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.712300    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21214503-bcc2-476a-9d26-c51d07d421df" (UID: "21214503-bcc2-476a-9d26-c51d07d421df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:49:50.712320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.712302    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-istio-token" (OuterVolumeSpecName: "istio-token") pod "21214503-bcc2-476a-9d26-c51d07d421df" (UID: "21214503-bcc2-476a-9d26-c51d07d421df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:49:50.712664 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.712647    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21214503-bcc2-476a-9d26-c51d07d421df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21214503-bcc2-476a-9d26-c51d07d421df" (UID: "21214503-bcc2-476a-9d26-c51d07d421df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:49:50.811759 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.811742    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21214503-bcc2-476a-9d26-c51d07d421df-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:50.811857 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.811760    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv2hn\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-kube-api-access-bv2hn\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:50.811857 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.811770    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21214503-bcc2-476a-9d26-c51d07d421df-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:50.811857 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.811780    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21214503-bcc2-476a-9d26-c51d07d421df-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:49:50.815760 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.815740    2576 generic.go:356] "Generic (PLEG): container finished" podID="21214503-bcc2-476a-9d26-c51d07d421df" containerID="d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621" exitCode=0
Sep 17 00:49:50.815854 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.815785    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"
Sep 17 00:49:50.815854 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.815831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" event={"ID":"21214503-bcc2-476a-9d26-c51d07d421df","Type":"ContainerDied","Data":"d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621"}
Sep 17 00:49:50.815958 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.815893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx" event={"ID":"21214503-bcc2-476a-9d26-c51d07d421df","Type":"ContainerDied","Data":"cfc2bafa38c2001e91ff6fc14084da615c1cd5444ad10189e1af77ca5dc46d65"}
Sep 17 00:49:50.815958 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.815925    2576 scope.go:117] "RemoveContainer" containerID="d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621"
Sep 17 00:49:50.825525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.825508    2576 scope.go:117] "RemoveContainer" containerID="66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0"
Sep 17 00:49:50.833898 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.833870    2576 scope.go:117] "RemoveContainer" containerID="d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621"
Sep 17 00:49:50.834540 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:49:50.834403    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621\": container with ID starting with d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621 not found: ID does not exist" containerID="d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621"
Sep 17 00:49:50.834540 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.834443    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621"} err="failed to get container status \"d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621\": rpc error: code = NotFound desc = could not find container \"d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621\": container with ID starting with d2bc59a03581303fd9a980926b4888e456ce667468b6f64076ba047dd0502621 not found: ID does not exist"
Sep 17 00:49:50.834540 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.834462    2576 scope.go:117] "RemoveContainer" containerID="66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0"
Sep 17 00:49:50.834789 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:49:50.834771    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0\": container with ID starting with 66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0 not found: ID does not exist" containerID="66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0"
Sep 17 00:49:50.834880 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.834794    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0"} err="failed to get container status \"66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0\": rpc error: code = NotFound desc = could not find container \"66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0\": container with ID starting with 66e5f1bfc25cb34fed720559bd79907fec94f21dbf6dd1f0e03e461c8febe0d0 not found: ID does not exist"
Sep 17 00:49:50.836875 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.836855    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"]
Sep 17 00:49:50.841129 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:50.841105    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1035c6c0-openshift-default-54b6fdcc5d-jkhcx"]
Sep 17 00:49:51.048567 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:51.048532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:51.662519 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:51.662485    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:51.763835 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:51.763798    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:51.865435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:51.865401    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:52.210026 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.209996    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:52.309029 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.308989    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:49:52.313517 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.313492    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"]
Sep 17 00:49:52.313920 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.313907    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy"
Sep 17 00:49:52.313966 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.313921    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy"
Sep 17 00:49:52.314013 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.314004    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy"
Sep 17 00:49:52.314051 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.314015    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy"
Sep 17 00:49:52.317093 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.317078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:49:52.321663 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.321645    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-39a5d066\""
Sep 17 00:49:52.321755 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.321687    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-39a5d066-dockercfg-5xwj5\""
Sep 17 00:49:52.329798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.329779    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"]
Sep 17 00:49:52.342520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.342498    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"]
Sep 17 00:49:52.342904 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.342890    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy"
Sep 17 00:49:52.342904 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.342904    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21214503-bcc2-476a-9d26-c51d07d421df" containerName="istio-proxy"
Sep 17 00:49:52.346067 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.346042    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:49:52.355379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.355359    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"]
Sep 17 00:49:52.426265 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.426224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c839283-0393-4c0f-a810-2d02288abf63-plugins-config-volume\") pod \"payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9\" (UID: \"6c839283-0393-4c0f-a810-2d02288abf63\") " pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:49:52.426444 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.426351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfmt\" (UniqueName: \"kubernetes.io/projected/4d250344-35b0-4212-9b98-d4ff3694fdf2-kube-api-access-wkfmt\") pod \"payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27\" (UID: \"4d250344-35b0-4212-9b98-d4ff3694fdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:49:52.426444 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.426389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjlv\" (UniqueName: \"kubernetes.io/projected/6c839283-0393-4c0f-a810-2d02288abf63-kube-api-access-gmjlv\") pod \"payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9\" (UID: \"6c839283-0393-4c0f-a810-2d02288abf63\") " pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:49:52.426444 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.426419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d250344-35b0-4212-9b98-d4ff3694fdf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27\" (UID: \"4d250344-35b0-4212-9b98-d4ff3694fdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:49:52.527275 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.527251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c839283-0393-4c0f-a810-2d02288abf63-plugins-config-volume\") pod \"payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9\" (UID: \"6c839283-0393-4c0f-a810-2d02288abf63\") " pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:49:52.527418 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.527321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfmt\" (UniqueName: \"kubernetes.io/projected/4d250344-35b0-4212-9b98-d4ff3694fdf2-kube-api-access-wkfmt\") pod \"payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27\" (UID: \"4d250344-35b0-4212-9b98-d4ff3694fdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:49:52.527418 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.527353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjlv\" (UniqueName: \"kubernetes.io/projected/6c839283-0393-4c0f-a810-2d02288abf63-kube-api-access-gmjlv\") pod \"payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9\" (UID: \"6c839283-0393-4c0f-a810-2d02288abf63\") " pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:49:52.527418 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.527378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d250344-35b0-4212-9b98-d4ff3694fdf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27\" (UID: \"4d250344-35b0-4212-9b98-d4ff3694fdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:49:52.528174 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.528132    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c839283-0393-4c0f-a810-2d02288abf63-plugins-config-volume\") pod \"payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9\" (UID: \"6c839283-0393-4c0f-a810-2d02288abf63\") " pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:49:52.528215 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.528197    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d250344-35b0-4212-9b98-d4ff3694fdf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27\" (UID: \"4d250344-35b0-4212-9b98-d4ff3694fdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:49:52.535746 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.535720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfmt\" (UniqueName: \"kubernetes.io/projected/4d250344-35b0-4212-9b98-d4ff3694fdf2-kube-api-access-wkfmt\") pod \"payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27\" (UID: \"4d250344-35b0-4212-9b98-d4ff3694fdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:49:52.535872 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.535810    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjlv\" (UniqueName: \"kubernetes.io/projected/6c839283-0393-4c0f-a810-2d02288abf63-kube-api-access-gmjlv\") pod \"payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9\" (UID: \"6c839283-0393-4c0f-a810-2d02288abf63\") " pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:49:52.588204 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.588178    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21214503-bcc2-476a-9d26-c51d07d421df" path="/var/lib/kubelet/pods/21214503-bcc2-476a-9d26-c51d07d421df/volumes"
Sep 17 00:49:52.628077 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.628057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:49:52.657267 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.657225    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:49:52.905776 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.905752    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"]
Sep 17 00:49:52.907176 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:49:52.907127    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d250344_35b0_4212_9b98_d4ff3694fdf2.slice/crio-cd5638751f44ca32e5412c597ab090c1cef77a3ada115759c05a72e01cd6d309 WatchSource:0}: Error finding container cd5638751f44ca32e5412c597ab090c1cef77a3ada115759c05a72e01cd6d309: Status 404 returned error can't find the container with id cd5638751f44ca32e5412c597ab090c1cef77a3ada115759c05a72e01cd6d309
Sep 17 00:49:52.947831 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:52.947796    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"]
Sep 17 00:49:52.962295 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:49:52.962264    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c839283_0393_4c0f_a810_2d02288abf63.slice/crio-8e0eeb4a18b636b247927506bb54e10e611d8e665f7acf344eb5771043b3893f WatchSource:0}: Error finding container 8e0eeb4a18b636b247927506bb54e10e611d8e665f7acf344eb5771043b3893f: Status 404 returned error can't find the container with id 8e0eeb4a18b636b247927506bb54e10e611d8e665f7acf344eb5771043b3893f
Sep 17 00:49:53.836256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:53.836125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9" event={"ID":"6c839283-0393-4c0f-a810-2d02288abf63","Type":"ContainerStarted","Data":"6b7b69259f3e36fd5f0092f34cfa3b108db82e2845c07920e7160ed38f002d10"}
Sep 17 00:49:53.836256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:53.836217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9" event={"ID":"6c839283-0393-4c0f-a810-2d02288abf63","Type":"ContainerStarted","Data":"8e0eeb4a18b636b247927506bb54e10e611d8e665f7acf344eb5771043b3893f"}
Sep 17 00:49:53.836513 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:53.836362    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:49:53.838315 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:53.838291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27" event={"ID":"4d250344-35b0-4212-9b98-d4ff3694fdf2","Type":"ContainerStarted","Data":"f94dae59169b50407c5e6c4cb4922f282e14fd1b3464905d893fc542d843151b"}
Sep 17 00:49:53.838429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:53.838337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27" event={"ID":"4d250344-35b0-4212-9b98-d4ff3694fdf2","Type":"ContainerStarted","Data":"cd5638751f44ca32e5412c597ab090c1cef77a3ada115759c05a72e01cd6d309"}
Sep 17 00:49:53.838588 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:53.838535    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:49:53.853167 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:53.853114    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9" podStartSLOduration=1.853101626 podStartE2EDuration="1.853101626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:49:53.850097432 +0000 UTC m=+1189.805693778" watchObservedRunningTime="2026-09-17 00:49:53.853101626 +0000 UTC m=+1189.808697967"
Sep 17 00:49:53.865521 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:49:53.865480    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27" podStartSLOduration=1.8654687490000001 podStartE2EDuration="1.865468749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:49:53.864130225 +0000 UTC m=+1189.819726569" watchObservedRunningTime="2026-09-17 00:49:53.865468749 +0000 UTC m=+1189.821065089"
Sep 17 00:50:04.847912 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:04.847878    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:50:04.848860 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:04.848833    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:50:05.010727 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:05.010644    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:50:05.012254 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:05.012232    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:50:06.703429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:06.703396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:08.932520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:08.932442    2576 generic.go:356] "Generic (PLEG): container finished" podID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerID="e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c" exitCode=0
Sep 17 00:50:08.932520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:08.932497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" event={"ID":"2eab41c5-983b-41ee-addb-43fbead3b0cf","Type":"ContainerDied","Data":"e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c"}
Sep 17 00:50:08.932896 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:08.932886    2576 scope.go:117] "RemoveContainer" containerID="e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c"
Sep 17 00:50:08.934096 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:08.934058    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:50:08.934210 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:08.934132    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:50:08.934210 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:08.934193    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:50:09.941341 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:09.941299    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" event={"ID":"2eab41c5-983b-41ee-addb-43fbead3b0cf","Type":"ContainerStarted","Data":"22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9"}
Sep 17 00:50:10.399628 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:10.399598    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:50:10.399812 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:10.399795    2576 patch_prober.go:28] interesting pod/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 00:50:10.399872 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:10.399838    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 00:50:11.399777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:11.399742    2576 patch_prober.go:28] interesting pod/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 00:50:11.400181 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:11.399811    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 00:50:12.400278 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:12.400241    2576 patch_prober.go:28] interesting pod/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 00:50:12.400663 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:12.400301    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 00:50:13.400315 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:13.400278    2576 patch_prober.go:28] interesting pod/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 00:50:13.400707 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:13.400340    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 00:50:13.528976 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:13.528945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:14.384522 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:14.384486    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:14.399575 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:14.399543    2576 patch_prober.go:28] interesting pod/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 00:50:14.399692 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:14.399612    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 00:50:14.991373 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:14.991327    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:15.404942 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:15.404904    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:50:15.405306 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:15.405289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:50:15.405906 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:15.405890    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:50:15.724537 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:15.724439    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:16.348608 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:16.348566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:17.035639 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:17.035608    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:17.766925 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:17.766889    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:18.140492 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.140453    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"]
Sep 17 00:50:18.145485 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.145446    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"
Sep 17 00:50:18.150713 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.150684    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:50:18.151461 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.151436    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"]
Sep 17 00:50:18.173764 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.173722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39a5d066-87wpv\" (UID: \"9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"
Sep 17 00:50:18.274968 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.274939    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39a5d066-87wpv\" (UID: \"9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"
Sep 17 00:50:18.275512 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.275494    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39a5d066-87wpv\" (UID: \"9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"
Sep 17 00:50:18.461820 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.461763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"
Sep 17 00:50:18.559562 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.559532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:18.732065 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:18.732038    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"]
Sep 17 00:50:18.734119 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:50:18.734086    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b35f6d5_9f3c_41b3_a1f5_db8ffd8afc12.slice/crio-3be16252ed87c2c58ea0c0f892468712c77f5eea9576e152c9b14b6409109b75 WatchSource:0}: Error finding container 3be16252ed87c2c58ea0c0f892468712c77f5eea9576e152c9b14b6409109b75: Status 404 returned error can't find the container with id 3be16252ed87c2c58ea0c0f892468712c77f5eea9576e152c9b14b6409109b75
Sep 17 00:50:19.005405 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:19.005341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv" event={"ID":"9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12","Type":"ContainerStarted","Data":"3be16252ed87c2c58ea0c0f892468712c77f5eea9576e152c9b14b6409109b75"}
Sep 17 00:50:19.264880 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:19.264811    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:19.890689 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:19.890648    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:20.012933 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:20.012900    2576 generic.go:356] "Generic (PLEG): container finished" podID="9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12" containerID="8d51db2a57367f6c9737580dc6a1c659ea06dd7923df9479fc8ae34ed2196cb1" exitCode=0
Sep 17 00:50:20.013324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:20.012954    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv" event={"ID":"9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12","Type":"ContainerDied","Data":"8d51db2a57367f6c9737580dc6a1c659ea06dd7923df9479fc8ae34ed2196cb1"}
Sep 17 00:50:20.598246 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:20.598217    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:21.151423 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:21.151396    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"
Sep 17 00:50:21.197250 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:21.197221    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12-maas-api-service-ca\") pod \"9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12\" (UID: \"9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12\") "
Sep 17 00:50:21.197527 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:21.197506    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12" (UID: "9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:50:21.296782 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:21.296752    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:21.298303 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:21.298281    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:21.929445 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:21.929411    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:22.024737 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:22.024704    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv" event={"ID":"9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12","Type":"ContainerDied","Data":"3be16252ed87c2c58ea0c0f892468712c77f5eea9576e152c9b14b6409109b75"}
Sep 17 00:50:22.024737 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:22.024741    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"
Sep 17 00:50:22.024938 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:22.024750    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3be16252ed87c2c58ea0c0f892468712c77f5eea9576e152c9b14b6409109b75"
Sep 17 00:50:22.658716 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:22.658686    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:27.178637 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:27.178612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:29.840865 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:29.840768    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:29.871054 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:29.871026    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"]
Sep 17 00:50:29.871310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:29.871282    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9" podUID="6c839283-0393-4c0f-a810-2d02288abf63" containerName="payload-processing" containerID="cri-o://6b7b69259f3e36fd5f0092f34cfa3b108db82e2845c07920e7160ed38f002d10" gracePeriod=30
Sep 17 00:50:29.880750 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:29.880720    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"]
Sep 17 00:50:29.881139 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:29.880968    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27" podUID="4d250344-35b0-4212-9b98-d4ff3694fdf2" containerName="payload-pre-processing" containerID="cri-o://f94dae59169b50407c5e6c4cb4922f282e14fd1b3464905d893fc542d843151b" gracePeriod=30
Sep 17 00:50:29.912029 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:29.912000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:29.980025 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:29.979990    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:30.033970 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.033908    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"]
Sep 17 00:50:30.034127 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.034062    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:30.035573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.035550    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39a5d066-87wpv"]
Sep 17 00:50:30.072566 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.072316    2576 generic.go:356] "Generic (PLEG): container finished" podID="4d250344-35b0-4212-9b98-d4ff3694fdf2" containerID="f94dae59169b50407c5e6c4cb4922f282e14fd1b3464905d893fc542d843151b" exitCode=0
Sep 17 00:50:30.072566 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.072455    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27" event={"ID":"4d250344-35b0-4212-9b98-d4ff3694fdf2","Type":"ContainerDied","Data":"f94dae59169b50407c5e6c4cb4922f282e14fd1b3464905d893fc542d843151b"}
Sep 17 00:50:30.074671 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.074650    2576 generic.go:356] "Generic (PLEG): container finished" podID="6c839283-0393-4c0f-a810-2d02288abf63" containerID="6b7b69259f3e36fd5f0092f34cfa3b108db82e2845c07920e7160ed38f002d10" exitCode=0
Sep 17 00:50:30.074775 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.074749    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9" event={"ID":"6c839283-0393-4c0f-a810-2d02288abf63","Type":"ContainerDied","Data":"6b7b69259f3e36fd5f0092f34cfa3b108db82e2845c07920e7160ed38f002d10"}
Sep 17 00:50:30.094388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.094330    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:30.175634 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.175548    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:50:30.193956 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.193735    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:50:30.195434 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.195403    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"]
Sep 17 00:50:30.195989 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.195937    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy" containerID="cri-o://22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9" gracePeriod=30
Sep 17 00:50:30.200620 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.200593    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:30.275068 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.275039    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkfmt\" (UniqueName: \"kubernetes.io/projected/4d250344-35b0-4212-9b98-d4ff3694fdf2-kube-api-access-wkfmt\") pod \"4d250344-35b0-4212-9b98-d4ff3694fdf2\" (UID: \"4d250344-35b0-4212-9b98-d4ff3694fdf2\") "
Sep 17 00:50:30.275245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.275085    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d250344-35b0-4212-9b98-d4ff3694fdf2-plugins-config-volume\") pod \"4d250344-35b0-4212-9b98-d4ff3694fdf2\" (UID: \"4d250344-35b0-4212-9b98-d4ff3694fdf2\") "
Sep 17 00:50:30.275317 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.275261    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmjlv\" (UniqueName: \"kubernetes.io/projected/6c839283-0393-4c0f-a810-2d02288abf63-kube-api-access-gmjlv\") pod \"6c839283-0393-4c0f-a810-2d02288abf63\" (UID: \"6c839283-0393-4c0f-a810-2d02288abf63\") "
Sep 17 00:50:30.275317 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.275290    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c839283-0393-4c0f-a810-2d02288abf63-plugins-config-volume\") pod \"6c839283-0393-4c0f-a810-2d02288abf63\" (UID: \"6c839283-0393-4c0f-a810-2d02288abf63\") "
Sep 17 00:50:30.275544 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.275513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d250344-35b0-4212-9b98-d4ff3694fdf2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d250344-35b0-4212-9b98-d4ff3694fdf2" (UID: "4d250344-35b0-4212-9b98-d4ff3694fdf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:50:30.275793 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.275744    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c839283-0393-4c0f-a810-2d02288abf63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c839283-0393-4c0f-a810-2d02288abf63" (UID: "6c839283-0393-4c0f-a810-2d02288abf63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:50:30.277328 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.277300    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d250344-35b0-4212-9b98-d4ff3694fdf2-kube-api-access-wkfmt" (OuterVolumeSpecName: "kube-api-access-wkfmt") pod "4d250344-35b0-4212-9b98-d4ff3694fdf2" (UID: "4d250344-35b0-4212-9b98-d4ff3694fdf2"). InnerVolumeSpecName "kube-api-access-wkfmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:50:30.277482 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.277460    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c839283-0393-4c0f-a810-2d02288abf63-kube-api-access-gmjlv" (OuterVolumeSpecName: "kube-api-access-gmjlv") pod "6c839283-0393-4c0f-a810-2d02288abf63" (UID: "6c839283-0393-4c0f-a810-2d02288abf63"). InnerVolumeSpecName "kube-api-access-gmjlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:50:30.376510 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.376414    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjlv\" (UniqueName: \"kubernetes.io/projected/6c839283-0393-4c0f-a810-2d02288abf63-kube-api-access-gmjlv\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:30.376510 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.376444    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c839283-0393-4c0f-a810-2d02288abf63-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:30.376510 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.376457    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkfmt\" (UniqueName: \"kubernetes.io/projected/4d250344-35b0-4212-9b98-d4ff3694fdf2-kube-api-access-wkfmt\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:30.376510 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.376470    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d250344-35b0-4212-9b98-d4ff3694fdf2-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:30.407707 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.407674    2576 patch_prober.go:28] interesting pod/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 17 00:50:30.407836 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.407737    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 00:50:30.570935 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.570910    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:30.589241 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.589216    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12" path="/var/lib/kubelet/pods/9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12/volumes"
Sep 17 00:50:30.954238 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:30.954213    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:31.045740 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.045709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:31.081547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.081524    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"
Sep 17 00:50:31.081547 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.081528    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27" event={"ID":"4d250344-35b0-4212-9b98-d4ff3694fdf2","Type":"ContainerDied","Data":"cd5638751f44ca32e5412c597ab090c1cef77a3ada115759c05a72e01cd6d309"}
Sep 17 00:50:31.081744 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.081592    2576 scope.go:117] "RemoveContainer" containerID="f94dae59169b50407c5e6c4cb4922f282e14fd1b3464905d893fc542d843151b"
Sep 17 00:50:31.083087 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.083060    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"
Sep 17 00:50:31.083250 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.083101    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9" event={"ID":"6c839283-0393-4c0f-a810-2d02288abf63","Type":"ContainerDied","Data":"8e0eeb4a18b636b247927506bb54e10e611d8e665f7acf344eb5771043b3893f"}
Sep 17 00:50:31.090780 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.090763    2576 scope.go:117] "RemoveContainer" containerID="6b7b69259f3e36fd5f0092f34cfa3b108db82e2845c07920e7160ed38f002d10"
Sep 17 00:50:31.103890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.103868    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"]
Sep 17 00:50:31.106662 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.106644    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39a5d066-786f59cdc7-r4jx9"]
Sep 17 00:50:31.112789 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.112750    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:31.114108 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.114087    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"]
Sep 17 00:50:31.117527 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.117509    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39a5d066-788dc99c98-54s27"]
Sep 17 00:50:31.463762 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:31.463732    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:32.260391 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:32.260352    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:32.588661 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:32.588631    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d250344-35b0-4212-9b98-d4ff3694fdf2" path="/var/lib/kubelet/pods/4d250344-35b0-4212-9b98-d4ff3694fdf2/volumes"
Sep 17 00:50:32.588994 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:32.588982    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c839283-0393-4c0f-a810-2d02288abf63" path="/var/lib/kubelet/pods/6c839283-0393-4c0f-a810-2d02288abf63/volumes"
Sep 17 00:50:32.691288 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:32.691263    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:32.991316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:32.991224    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:33.747810 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:33.747778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:34.353980 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:34.353946    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:35.461435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.461386    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:50:35.518973 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.518949    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-envoy\") pod \"2eab41c5-983b-41ee-addb-43fbead3b0cf\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") "
Sep 17 00:50:35.518973 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.518974    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-data\") pod \"2eab41c5-983b-41ee-addb-43fbead3b0cf\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") "
Sep 17 00:50:35.519178 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.518995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-podinfo\") pod \"2eab41c5-983b-41ee-addb-43fbead3b0cf\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") "
Sep 17 00:50:35.519178 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519062    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-certs\") pod \"2eab41c5-983b-41ee-addb-43fbead3b0cf\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") "
Sep 17 00:50:35.519178 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519080    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab41c5-983b-41ee-addb-43fbead3b0cf-istiod-ca-cert\") pod \"2eab41c5-983b-41ee-addb-43fbead3b0cf\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") "
Sep 17 00:50:35.519178 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519118    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-socket\") pod \"2eab41c5-983b-41ee-addb-43fbead3b0cf\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") "
Sep 17 00:50:35.519178 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519174    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-token\") pod \"2eab41c5-983b-41ee-addb-43fbead3b0cf\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") "
Sep 17 00:50:35.519440 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519199    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-credential-socket\") pod \"2eab41c5-983b-41ee-addb-43fbead3b0cf\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") "
Sep 17 00:50:35.519440 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519242    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2q55\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-kube-api-access-g2q55\") pod \"2eab41c5-983b-41ee-addb-43fbead3b0cf\" (UID: \"2eab41c5-983b-41ee-addb-43fbead3b0cf\") "
Sep 17 00:50:35.519440 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519274    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "2eab41c5-983b-41ee-addb-43fbead3b0cf" (UID: "2eab41c5-983b-41ee-addb-43fbead3b0cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:50:35.519440 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519397    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2eab41c5-983b-41ee-addb-43fbead3b0cf" (UID: "2eab41c5-983b-41ee-addb-43fbead3b0cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:50:35.519640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519450    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2eab41c5-983b-41ee-addb-43fbead3b0cf" (UID: "2eab41c5-983b-41ee-addb-43fbead3b0cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:50:35.519640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519596    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.519640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519638    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.519795 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519651    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.519795 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519651    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2eab41c5-983b-41ee-addb-43fbead3b0cf" (UID: "2eab41c5-983b-41ee-addb-43fbead3b0cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:50:35.519795 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.519687    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eab41c5-983b-41ee-addb-43fbead3b0cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2eab41c5-983b-41ee-addb-43fbead3b0cf" (UID: "2eab41c5-983b-41ee-addb-43fbead3b0cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:50:35.521471 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.521449    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-kube-api-access-g2q55" (OuterVolumeSpecName: "kube-api-access-g2q55") pod "2eab41c5-983b-41ee-addb-43fbead3b0cf" (UID: "2eab41c5-983b-41ee-addb-43fbead3b0cf"). InnerVolumeSpecName "kube-api-access-g2q55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:50:35.521590 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.521510    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2eab41c5-983b-41ee-addb-43fbead3b0cf" (UID: "2eab41c5-983b-41ee-addb-43fbead3b0cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:50:35.521728 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.521714    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "2eab41c5-983b-41ee-addb-43fbead3b0cf" (UID: "2eab41c5-983b-41ee-addb-43fbead3b0cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:50:35.521838 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.521821    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2eab41c5-983b-41ee-addb-43fbead3b0cf" (UID: "2eab41c5-983b-41ee-addb-43fbead3b0cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:50:35.620351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.620328    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.620351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.620349    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.620488 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.620359    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2q55\" (UniqueName: \"kubernetes.io/projected/2eab41c5-983b-41ee-addb-43fbead3b0cf-kube-api-access-g2q55\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.620488 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.620368    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.620488 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.620376    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eab41c5-983b-41ee-addb-43fbead3b0cf-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.620488 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:35.620383    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eab41c5-983b-41ee-addb-43fbead3b0cf-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:50:36.114667 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.114633    2576 generic.go:356] "Generic (PLEG): container finished" podID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerID="22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9" exitCode=0
Sep 17 00:50:36.114886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.114692    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"
Sep 17 00:50:36.114886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.114715    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" event={"ID":"2eab41c5-983b-41ee-addb-43fbead3b0cf","Type":"ContainerDied","Data":"22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9"}
Sep 17 00:50:36.114886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.114768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb" event={"ID":"2eab41c5-983b-41ee-addb-43fbead3b0cf","Type":"ContainerDied","Data":"7dbf902646f6a6bec74dc6e80ce5c837e29714402567c070d88b8c11f5c0e21c"}
Sep 17 00:50:36.114886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.114788    2576 scope.go:117] "RemoveContainer" containerID="22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9"
Sep 17 00:50:36.124810 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.124795    2576 scope.go:117] "RemoveContainer" containerID="e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c"
Sep 17 00:50:36.133529 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.133497    2576 scope.go:117] "RemoveContainer" containerID="22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9"
Sep 17 00:50:36.133777 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:50:36.133756    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9\": container with ID starting with 22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9 not found: ID does not exist" containerID="22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9"
Sep 17 00:50:36.133872 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.133787    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9"} err="failed to get container status \"22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9\": rpc error: code = NotFound desc = could not find container \"22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9\": container with ID starting with 22d94464580b4e34748fc4a49e8f7d1d5e9bae3a71f961a517c1647b2effd9a9 not found: ID does not exist"
Sep 17 00:50:36.133872 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.133811    2576 scope.go:117] "RemoveContainer" containerID="e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c"
Sep 17 00:50:36.134124 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:50:36.134105    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c\": container with ID starting with e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c not found: ID does not exist" containerID="e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c"
Sep 17 00:50:36.134212 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.134131    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c"} err="failed to get container status \"e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c\": rpc error: code = NotFound desc = could not find container \"e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c\": container with ID starting with e8c90816e4f934a80023a989b7a062960ac9af3848d88e042e079b1da952812c not found: ID does not exist"
Sep 17 00:50:36.135638 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.135619    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"]
Sep 17 00:50:36.139819 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.139798    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-39a5d066-openshift-default-6d9cc4486d-r6ndb"]
Sep 17 00:50:36.592612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:36.592584    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" path="/var/lib/kubelet/pods/2eab41c5-983b-41ee-addb-43fbead3b0cf/volumes"
Sep 17 00:50:38.709553 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:38.709527    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:38.817347 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:38.817320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:38.909841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:38.909813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:39.010200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:39.010099    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:39.490504 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:39.490469    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:59.596446 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.596358    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"]
Sep 17 00:50:59.596941 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.596925    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12" containerName="revoke-keys"
Sep 17 00:50:59.597010 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.596945    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12" containerName="revoke-keys"
Sep 17 00:50:59.597010 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.596967    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy"
Sep 17 00:50:59.597010 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.596976    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy"
Sep 17 00:50:59.597010 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.597001    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c839283-0393-4c0f-a810-2d02288abf63" containerName="payload-processing"
Sep 17 00:50:59.597198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.597012    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c839283-0393-4c0f-a810-2d02288abf63" containerName="payload-processing"
Sep 17 00:50:59.597198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.597025    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d250344-35b0-4212-9b98-d4ff3694fdf2" containerName="payload-pre-processing"
Sep 17 00:50:59.597198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.597034    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d250344-35b0-4212-9b98-d4ff3694fdf2" containerName="payload-pre-processing"
Sep 17 00:50:59.597198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.597132    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy"
Sep 17 00:50:59.597198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.597163    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b35f6d5-9f3c-41b3-a1f5-db8ffd8afc12" containerName="revoke-keys"
Sep 17 00:50:59.597198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.597173    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy"
Sep 17 00:50:59.597198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.597185    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c839283-0393-4c0f-a810-2d02288abf63" containerName="payload-processing"
Sep 17 00:50:59.597198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.597193    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d250344-35b0-4212-9b98-d4ff3694fdf2" containerName="payload-pre-processing"
Sep 17 00:50:59.602286 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.602266    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.605806 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.605772    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e491c5\"/\"kube-root-ca.crt\""
Sep 17 00:50:59.605937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.605772    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e491c5\"/\"openshift-service-ca.crt\""
Sep 17 00:50:59.605937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.605791    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e491c5\"/\"default-dockercfg-bqkbk\""
Sep 17 00:50:59.605937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.605792    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e491c5\"/\"test-model-e491c5-kserve-self-signed-certs\""
Sep 17 00:50:59.606085 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.606048    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"]
Sep 17 00:50:59.623558 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.623535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:50:59.708468 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.708442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8sj\" (UniqueName: \"kubernetes.io/projected/43100e01-42bb-40c1-98f7-17c3ae215f0b-kube-api-access-wf8sj\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.708604 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.708477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-home\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.708604 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.708573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-dshm\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.708721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.708605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-tmp-dir\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.708721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.708711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43100e01-42bb-40c1-98f7-17c3ae215f0b-tls-certs\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.708830 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.708728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-model-cache\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.809874 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.809844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-dshm\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.810052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.809889    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-tmp-dir\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.810052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.809926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43100e01-42bb-40c1-98f7-17c3ae215f0b-tls-certs\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.810052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.809950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-model-cache\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.810052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.810005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8sj\" (UniqueName: \"kubernetes.io/projected/43100e01-42bb-40c1-98f7-17c3ae215f0b-kube-api-access-wf8sj\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.810052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.810033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-home\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.810413 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.810358    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-tmp-dir\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.810533 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.810423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-model-cache\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.810533 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.810508    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-home\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.812337 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.812316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-dshm\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.812633 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.812616    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43100e01-42bb-40c1-98f7-17c3ae215f0b-tls-certs\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.817912 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.817890    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8sj\" (UniqueName: \"kubernetes.io/projected/43100e01-42bb-40c1-98f7-17c3ae215f0b-kube-api-access-wf8sj\") pod \"test-model-e491c5-kserve-7d949d77d7-vz6xb\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") " pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:50:59.912693 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:50:59.912642    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:51:00.183598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:00.183570    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"]
Sep 17 00:51:00.185662 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:51:00.185638    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43100e01_42bb_40c1_98f7_17c3ae215f0b.slice/crio-6f79084e3c750a8a370a7c491038e810afa44f5bef3168cf7a8ee968220ace58 WatchSource:0}: Error finding container 6f79084e3c750a8a370a7c491038e810afa44f5bef3168cf7a8ee968220ace58: Status 404 returned error can't find the container with id 6f79084e3c750a8a370a7c491038e810afa44f5bef3168cf7a8ee968220ace58
Sep 17 00:51:00.251370 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:00.251342    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb" event={"ID":"43100e01-42bb-40c1-98f7-17c3ae215f0b","Type":"ContainerStarted","Data":"1ad3b24a4368e9227bea1395a0362c48630709744ef80dd747aac87830c783f6"}
Sep 17 00:51:00.251501 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:00.251382    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb" event={"ID":"43100e01-42bb-40c1-98f7-17c3ae215f0b","Type":"ContainerStarted","Data":"6f79084e3c750a8a370a7c491038e810afa44f5bef3168cf7a8ee968220ace58"}
Sep 17 00:51:00.269825 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:00.269783    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb" podStartSLOduration=1.269767166 podStartE2EDuration="1.269767166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:51:00.267433379 +0000 UTC m=+1256.223029732" watchObservedRunningTime="2026-09-17 00:51:00.269767166 +0000 UTC m=+1256.225363505"
Sep 17 00:51:02.022258 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.022230    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"]
Sep 17 00:51:02.022684 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.022670    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy"
Sep 17 00:51:02.022730 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.022685    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eab41c5-983b-41ee-addb-43fbead3b0cf" containerName="istio-proxy"
Sep 17 00:51:02.025986 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.025970    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"
Sep 17 00:51:02.028424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.028403    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:51:02.032771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.032750    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"]
Sep 17 00:51:02.131443 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.131416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ad70b0-b040-42db-b5fc-5072a9ca63d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-6af38a-gj9p5\" (UID: \"48ad70b0-b040-42db-b5fc-5072a9ca63d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"
Sep 17 00:51:02.232776 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.232754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ad70b0-b040-42db-b5fc-5072a9ca63d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-6af38a-gj9p5\" (UID: \"48ad70b0-b040-42db-b5fc-5072a9ca63d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"
Sep 17 00:51:02.233338 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.233320    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ad70b0-b040-42db-b5fc-5072a9ca63d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-6af38a-gj9p5\" (UID: \"48ad70b0-b040-42db-b5fc-5072a9ca63d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"
Sep 17 00:51:02.338944 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.338922    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"
Sep 17 00:51:02.370360 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.370333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:02.610170 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:02.610126    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"]
Sep 17 00:51:02.611547 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:51:02.611517    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ad70b0_b040_42db_b5fc_5072a9ca63d6.slice/crio-f9d9b24dd3a2e267f1678c0ad961c352b6ca8446af613c26c689659ad9c4021e WatchSource:0}: Error finding container f9d9b24dd3a2e267f1678c0ad961c352b6ca8446af613c26c689659ad9c4021e: Status 404 returned error can't find the container with id f9d9b24dd3a2e267f1678c0ad961c352b6ca8446af613c26c689659ad9c4021e
Sep 17 00:51:03.048886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:03.048849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:03.268520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:03.268437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5" event={"ID":"48ad70b0-b040-42db-b5fc-5072a9ca63d6","Type":"ContainerStarted","Data":"b285be96f8ba445ebeb959e78638d22678b924dacc51fbce92f97e89269b9767"}
Sep 17 00:51:03.268520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:03.268496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5" event={"ID":"48ad70b0-b040-42db-b5fc-5072a9ca63d6","Type":"ContainerStarted","Data":"f9d9b24dd3a2e267f1678c0ad961c352b6ca8446af613c26c689659ad9c4021e"}
Sep 17 00:51:03.281821 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:03.281755    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5" podStartSLOduration=1.28173876 podStartE2EDuration="1.28173876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:51:03.281065449 +0000 UTC m=+1259.236661793" watchObservedRunningTime="2026-09-17 00:51:03.28173876 +0000 UTC m=+1259.237335099"
Sep 17 00:51:03.764105 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:03.764079    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:04.276559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:04.276525    2576 generic.go:356] "Generic (PLEG): container finished" podID="48ad70b0-b040-42db-b5fc-5072a9ca63d6" containerID="b285be96f8ba445ebeb959e78638d22678b924dacc51fbce92f97e89269b9767" exitCode=0
Sep 17 00:51:04.276961 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:04.276576    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5" event={"ID":"48ad70b0-b040-42db-b5fc-5072a9ca63d6","Type":"ContainerDied","Data":"b285be96f8ba445ebeb959e78638d22678b924dacc51fbce92f97e89269b9767"}
Sep 17 00:51:04.372743 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:04.372714    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:05.156999 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:05.156968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:05.409723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:05.409518    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:05.437959 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:05.437936    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"
Sep 17 00:51:05.567410 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:05.567382    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ad70b0-b040-42db-b5fc-5072a9ca63d6-maas-api-service-ca\") pod \"48ad70b0-b040-42db-b5fc-5072a9ca63d6\" (UID: \"48ad70b0-b040-42db-b5fc-5072a9ca63d6\") "
Sep 17 00:51:05.567737 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:05.567715    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ad70b0-b040-42db-b5fc-5072a9ca63d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48ad70b0-b040-42db-b5fc-5072a9ca63d6" (UID: "48ad70b0-b040-42db-b5fc-5072a9ca63d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:51:05.668621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:05.668559    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ad70b0-b040-42db-b5fc-5072a9ca63d6-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:05.791686 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:05.791654    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:06.288665 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:06.288635    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"
Sep 17 00:51:06.288852 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:06.288634    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5" event={"ID":"48ad70b0-b040-42db-b5fc-5072a9ca63d6","Type":"ContainerDied","Data":"f9d9b24dd3a2e267f1678c0ad961c352b6ca8446af613c26c689659ad9c4021e"}
Sep 17 00:51:06.288852 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:06.288775    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9d9b24dd3a2e267f1678c0ad961c352b6ca8446af613c26c689659ad9c4021e"
Sep 17 00:51:06.576876 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:06.576813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:09.913282 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:09.913251    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:51:09.913841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:09.913332    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:51:09.925361 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:09.925342    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:51:10.321036 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:10.321010    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:51:11.006421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:11.006393    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:11.063435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:11.063412    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:11.129992 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:11.129968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:11.230680 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:11.230657    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:12.463272 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:12.463238    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:12.524687 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:12.524659    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:12.581193 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:12.581157    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:12.635716 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:12.635683    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"]
Sep 17 00:51:12.637133 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:12.637106    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:12.639518 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:12.639494    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6af38a-gj9p5"]
Sep 17 00:51:12.688865 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:12.688845    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:13.031949 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:13.031912    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:13.122399 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:13.122373    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:13.739371 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:13.739344    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:14.446590 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:14.446552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:14.588647 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:14.588613    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ad70b0-b040-42db-b5fc-5072a9ca63d6" path="/var/lib/kubelet/pods/48ad70b0-b040-42db-b5fc-5072a9ca63d6/volumes"
Sep 17 00:51:15.069251 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:15.069220    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:15.749228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:15.749191    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:18.008662 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:18.008631    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:22.888010 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:22.887971    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:22.942522 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:22.942495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:23.313376 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:23.313271    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:25.257166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.257106    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:25.367182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.367135    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"]
Sep 17 00:51:25.367602 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.367589    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ad70b0-b040-42db-b5fc-5072a9ca63d6" containerName="revoke-keys"
Sep 17 00:51:25.367659 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.367604    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ad70b0-b040-42db-b5fc-5072a9ca63d6" containerName="revoke-keys"
Sep 17 00:51:25.367698 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.367673    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ad70b0-b040-42db-b5fc-5072a9ca63d6" containerName="revoke-keys"
Sep 17 00:51:25.371234 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.371218    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"
Sep 17 00:51:25.373572 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.373549    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:51:25.376035 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.376008    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"]
Sep 17 00:51:25.444887 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.444854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd8d31-6eec-4792-9b9e-d4fb3f25434a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f40754-zt285\" (UID: \"79fd8d31-6eec-4792-9b9e-d4fb3f25434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"
Sep 17 00:51:25.476162 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.476117    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:25.545765 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.545661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd8d31-6eec-4792-9b9e-d4fb3f25434a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f40754-zt285\" (UID: \"79fd8d31-6eec-4792-9b9e-d4fb3f25434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"
Sep 17 00:51:25.546456 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.546430    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd8d31-6eec-4792-9b9e-d4fb3f25434a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f40754-zt285\" (UID: \"79fd8d31-6eec-4792-9b9e-d4fb3f25434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"
Sep 17 00:51:25.683942 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.683916    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"
Sep 17 00:51:25.953817 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:25.953787    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"]
Sep 17 00:51:25.955362 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:51:25.955335    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fd8d31_6eec_4792_9b9e_d4fb3f25434a.slice/crio-aa8e546ba08068538c1dd7d5e76fc91beab9b567f847fe3b3f196b9b630e820c WatchSource:0}: Error finding container aa8e546ba08068538c1dd7d5e76fc91beab9b567f847fe3b3f196b9b630e820c: Status 404 returned error can't find the container with id aa8e546ba08068538c1dd7d5e76fc91beab9b567f847fe3b3f196b9b630e820c
Sep 17 00:51:26.246300 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:26.246275    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:26.405673 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:26.405636    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285" event={"ID":"79fd8d31-6eec-4792-9b9e-d4fb3f25434a","Type":"ContainerStarted","Data":"532b835f2c07a1f54f8f5d516e78f61d9928969e8e9a74bb0832123054d3bec2"}
Sep 17 00:51:26.406135 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:26.405683    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285" event={"ID":"79fd8d31-6eec-4792-9b9e-d4fb3f25434a","Type":"ContainerStarted","Data":"aa8e546ba08068538c1dd7d5e76fc91beab9b567f847fe3b3f196b9b630e820c"}
Sep 17 00:51:26.420054 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:26.420011    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285" podStartSLOduration=1.419996949 podStartE2EDuration="1.419996949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:51:26.418249028 +0000 UTC m=+1282.373845369" watchObservedRunningTime="2026-09-17 00:51:26.419996949 +0000 UTC m=+1282.375593318"
Sep 17 00:51:27.005296 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:27.005212    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:27.103501 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:27.103466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:27.412681 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:27.412653    2576 generic.go:356] "Generic (PLEG): container finished" podID="79fd8d31-6eec-4792-9b9e-d4fb3f25434a" containerID="532b835f2c07a1f54f8f5d516e78f61d9928969e8e9a74bb0832123054d3bec2" exitCode=0
Sep 17 00:51:27.413058 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:27.412696    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285" event={"ID":"79fd8d31-6eec-4792-9b9e-d4fb3f25434a","Type":"ContainerDied","Data":"532b835f2c07a1f54f8f5d516e78f61d9928969e8e9a74bb0832123054d3bec2"}
Sep 17 00:51:27.756953 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:27.756857    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:28.377014 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:28.376984    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:28.564406 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:28.564383    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"
Sep 17 00:51:28.573813 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:28.573793    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd8d31-6eec-4792-9b9e-d4fb3f25434a-maas-api-service-ca\") pod \"79fd8d31-6eec-4792-9b9e-d4fb3f25434a\" (UID: \"79fd8d31-6eec-4792-9b9e-d4fb3f25434a\") "
Sep 17 00:51:28.574103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:28.574083    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fd8d31-6eec-4792-9b9e-d4fb3f25434a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79fd8d31-6eec-4792-9b9e-d4fb3f25434a" (UID: "79fd8d31-6eec-4792-9b9e-d4fb3f25434a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:51:28.675454 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:28.675392    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79fd8d31-6eec-4792-9b9e-d4fb3f25434a-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:29.073881 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:29.073845    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:29.424829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:29.424744    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"
Sep 17 00:51:29.424829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:29.424785    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285" event={"ID":"79fd8d31-6eec-4792-9b9e-d4fb3f25434a","Type":"ContainerDied","Data":"aa8e546ba08068538c1dd7d5e76fc91beab9b567f847fe3b3f196b9b630e820c"}
Sep 17 00:51:29.425017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:29.424854    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8e546ba08068538c1dd7d5e76fc91beab9b567f847fe3b3f196b9b630e820c"
Sep 17 00:51:30.295033 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.295001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:30.296797 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.296775    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"]
Sep 17 00:51:30.297205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.297192    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fd8d31-6eec-4792-9b9e-d4fb3f25434a" containerName="revoke-keys"
Sep 17 00:51:30.297257 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.297207    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fd8d31-6eec-4792-9b9e-d4fb3f25434a" containerName="revoke-keys"
Sep 17 00:51:30.297292 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.297268    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fd8d31-6eec-4792-9b9e-d4fb3f25434a" containerName="revoke-keys"
Sep 17 00:51:30.300384 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.300371    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.303713 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.303690    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"default-dockercfg-lqgbh\""
Sep 17 00:51:30.303837 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.303722    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"premium-w0-a79f56-kserve-self-signed-certs\""
Sep 17 00:51:30.303837 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.303753    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"openshift-service-ca.crt\""
Sep 17 00:51:30.303837 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.303722    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"kube-root-ca.crt\""
Sep 17 00:51:30.308358 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.308332    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"]
Sep 17 00:51:30.383496 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.383465    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:30.495514 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.495482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tls-certs\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.495697 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.495521    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-home\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.495697 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.495568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-dshm\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.495697 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.495596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44672\" (UniqueName: \"kubernetes.io/projected/e4110dd9-03ae-4f71-b4bc-bb08c9951629-kube-api-access-44672\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.495697 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.495666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-model-cache\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.495697 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.495679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tmp-dir\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.596777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.596705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-dshm\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.596777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.596742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44672\" (UniqueName: \"kubernetes.io/projected/e4110dd9-03ae-4f71-b4bc-bb08c9951629-kube-api-access-44672\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.596975 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.596792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-model-cache\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.596975 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.596810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tmp-dir\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.596975 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.596840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tls-certs\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.596975 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.596874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-home\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.597224 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.597200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tmp-dir\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.597285 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.597244    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-model-cache\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.597325 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.597304    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-home\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.599185 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.599166    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-dshm\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.599461 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.599445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tls-certs\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.604205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.604182    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44672\" (UniqueName: \"kubernetes.io/projected/e4110dd9-03ae-4f71-b4bc-bb08c9951629-kube-api-access-44672\") pod \"premium-w0-a79f56-kserve-5cc7c85756-b4szt\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") " pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.610059 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.610037    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:30.888992 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:30.888961    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"]
Sep 17 00:51:30.890699 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:51:30.890672    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4110dd9_03ae_4f71_b4bc_bb08c9951629.slice/crio-2ca3e0f6bb6202fbf85e2d999a65ae9583b2722fbdb026206893479a0e90baf5 WatchSource:0}: Error finding container 2ca3e0f6bb6202fbf85e2d999a65ae9583b2722fbdb026206893479a0e90baf5: Status 404 returned error can't find the container with id 2ca3e0f6bb6202fbf85e2d999a65ae9583b2722fbdb026206893479a0e90baf5
Sep 17 00:51:31.437663 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:31.437624    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt" event={"ID":"e4110dd9-03ae-4f71-b4bc-bb08c9951629","Type":"ContainerStarted","Data":"9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416"}
Sep 17 00:51:31.438055 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:31.437676    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt" event={"ID":"e4110dd9-03ae-4f71-b4bc-bb08c9951629","Type":"ContainerStarted","Data":"2ca3e0f6bb6202fbf85e2d999a65ae9583b2722fbdb026206893479a0e90baf5"}
Sep 17 00:51:31.455004 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:31.454952    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt" podStartSLOduration=1.454936917 podStartE2EDuration="1.454936917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:51:31.453330649 +0000 UTC m=+1287.408926983" watchObservedRunningTime="2026-09-17 00:51:31.454936917 +0000 UTC m=+1287.410533258"
Sep 17 00:51:32.643974 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:32.643940    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:32.703943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:32.703909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:32.779334 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:32.779294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:32.880806 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:32.880768    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:34.851820 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:34.851767    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:34.953081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:34.953039    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:35.874579 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:35.874540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:35.904334 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:35.904305    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"]
Sep 17 00:51:35.904550 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:35.904526    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4" podUID="ba12a105-636d-4d55-b5b7-b15ca90b5453" containerName="payload-processing" containerID="cri-o://1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73" gracePeriod=30
Sep 17 00:51:35.974704 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:35.974653    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:36.070957 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.070929    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"]
Sep 17 00:51:36.074594 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.074558    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:36.075407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.075388    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f40754-zt285"]
Sep 17 00:51:36.135039 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.134958    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:36.169979 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.169954    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:51:36.249548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.249510    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba12a105-636d-4d55-b5b7-b15ca90b5453-plugins-config-volume\") pod \"ba12a105-636d-4d55-b5b7-b15ca90b5453\" (UID: \"ba12a105-636d-4d55-b5b7-b15ca90b5453\") "
Sep 17 00:51:36.249744 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.249682    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbh78\" (UniqueName: \"kubernetes.io/projected/ba12a105-636d-4d55-b5b7-b15ca90b5453-kube-api-access-kbh78\") pod \"ba12a105-636d-4d55-b5b7-b15ca90b5453\" (UID: \"ba12a105-636d-4d55-b5b7-b15ca90b5453\") "
Sep 17 00:51:36.249948 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.249922    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba12a105-636d-4d55-b5b7-b15ca90b5453-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba12a105-636d-4d55-b5b7-b15ca90b5453" (UID: "ba12a105-636d-4d55-b5b7-b15ca90b5453"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:51:36.252029 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.251998    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba12a105-636d-4d55-b5b7-b15ca90b5453-kube-api-access-kbh78" (OuterVolumeSpecName: "kube-api-access-kbh78") pod "ba12a105-636d-4d55-b5b7-b15ca90b5453" (UID: "ba12a105-636d-4d55-b5b7-b15ca90b5453"). InnerVolumeSpecName "kube-api-access-kbh78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:51:36.350585 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.350546    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbh78\" (UniqueName: \"kubernetes.io/projected/ba12a105-636d-4d55-b5b7-b15ca90b5453-kube-api-access-kbh78\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:36.350585 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.350578    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba12a105-636d-4d55-b5b7-b15ca90b5453-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:36.406631 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.406508    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:36.477069 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.477032    2576 generic.go:356] "Generic (PLEG): container finished" podID="ba12a105-636d-4d55-b5b7-b15ca90b5453" containerID="1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73" exitCode=0
Sep 17 00:51:36.477069 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.477074    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"
Sep 17 00:51:36.477315 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.477106    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4" event={"ID":"ba12a105-636d-4d55-b5b7-b15ca90b5453","Type":"ContainerDied","Data":"1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73"}
Sep 17 00:51:36.477315 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.477179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4" event={"ID":"ba12a105-636d-4d55-b5b7-b15ca90b5453","Type":"ContainerDied","Data":"8415044902d64c6371fa3c43a787b2424ce260a38d2ba691d54051946d321a9d"}
Sep 17 00:51:36.477315 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.477203    2576 scope.go:117] "RemoveContainer" containerID="1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73"
Sep 17 00:51:36.488880 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.487233    2576 scope.go:117] "RemoveContainer" containerID="1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73"
Sep 17 00:51:36.489400 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:51:36.489371    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73\": container with ID starting with 1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73 not found: ID does not exist" containerID="1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73"
Sep 17 00:51:36.489474 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.489412    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73"} err="failed to get container status \"1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73\": rpc error: code = NotFound desc = could not find container \"1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73\": container with ID starting with 1764a7caa42cdbd020dad7dfa6a4bd576adf19ca325125864824f64015370b73 not found: ID does not exist"
Sep 17 00:51:36.498780 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.498753    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"]
Sep 17 00:51:36.500971 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.500928    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:36.502264 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.502243    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f40754-65fc6f54c9-9gkh4"]
Sep 17 00:51:36.589918 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.589878    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79fd8d31-6eec-4792-9b9e-d4fb3f25434a" path="/var/lib/kubelet/pods/79fd8d31-6eec-4792-9b9e-d4fb3f25434a/volumes"
Sep 17 00:51:36.590309 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:36.590293    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba12a105-636d-4d55-b5b7-b15ca90b5453" path="/var/lib/kubelet/pods/ba12a105-636d-4d55-b5b7-b15ca90b5453/volumes"
Sep 17 00:51:37.256762 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:37.256702    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:37.904410 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:37.904377    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:38.582944 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:38.582901    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:40.610899 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:40.610868    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:40.611317 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:40.610911    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:40.623968 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:40.623943    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:41.518745 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:41.518716    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:51:42.230473 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:42.230443    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:42.284695 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:42.284662    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:42.462672 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:42.462582    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"]
Sep 17 00:51:42.462915 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:42.462890    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy" containerID="cri-o://f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358" gracePeriod=30
Sep 17 00:51:42.476899 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:42.476876    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:43.154958 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:43.154914    2576 patch_prober.go:28] interesting pod/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 00:51:43.155124 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:43.154987    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 00:51:43.308458 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:43.308423    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:44.090836 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:44.090808    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:47.729516 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.729455    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:51:47.756348 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.756319    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-podinfo\") pod \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") "
Sep 17 00:51:47.756487 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.756418    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istiod-ca-cert\") pod \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") "
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.756680    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-certs\") pod \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") "
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.756721    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-socket\") pod \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") "
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.756780    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-credential-socket\") pod \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") "
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.756822    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-data\") pod \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") "
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.756857    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-token\") pod \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") "
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.756893    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhw76\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-kube-api-access-rhw76\") pod \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") "
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.756951    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-envoy\") pod \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\" (UID: \"1a97aaa1-df48-41b0-a7a3-830e8bbef608\") "
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.757094    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a97aaa1-df48-41b0-a7a3-830e8bbef608" (UID: "1a97aaa1-df48-41b0-a7a3-830e8bbef608"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.757114    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a97aaa1-df48-41b0-a7a3-830e8bbef608" (UID: "1a97aaa1-df48-41b0-a7a3-830e8bbef608"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.757115    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a97aaa1-df48-41b0-a7a3-830e8bbef608" (UID: "1a97aaa1-df48-41b0-a7a3-830e8bbef608"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.757384    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.757403    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.757417    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.757606    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a97aaa1-df48-41b0-a7a3-830e8bbef608" (UID: "1a97aaa1-df48-41b0-a7a3-830e8bbef608"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:51:47.757658 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.757653    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a97aaa1-df48-41b0-a7a3-830e8bbef608" (UID: "1a97aaa1-df48-41b0-a7a3-830e8bbef608"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:51:47.759519 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.759240    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a97aaa1-df48-41b0-a7a3-830e8bbef608" (UID: "1a97aaa1-df48-41b0-a7a3-830e8bbef608"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:51:47.759760 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.759735    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-kube-api-access-rhw76" (OuterVolumeSpecName: "kube-api-access-rhw76") pod "1a97aaa1-df48-41b0-a7a3-830e8bbef608" (UID: "1a97aaa1-df48-41b0-a7a3-830e8bbef608"). InnerVolumeSpecName "kube-api-access-rhw76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:51:47.760263 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.760238    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a97aaa1-df48-41b0-a7a3-830e8bbef608" (UID: "1a97aaa1-df48-41b0-a7a3-830e8bbef608"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:51:47.761010 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.760990    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a97aaa1-df48-41b0-a7a3-830e8bbef608" (UID: "1a97aaa1-df48-41b0-a7a3-830e8bbef608"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:51:47.858207 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.858174    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:47.858207 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.858196    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:47.858207 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.858206    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:47.858207 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.858214    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhw76\" (UniqueName: \"kubernetes.io/projected/1a97aaa1-df48-41b0-a7a3-830e8bbef608-kube-api-access-rhw76\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:47.858506 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.858223    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:47.858506 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:47.858234    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a97aaa1-df48-41b0-a7a3-830e8bbef608-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:51:48.548636 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.548600    2576 generic.go:356] "Generic (PLEG): container finished" podID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerID="f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358" exitCode=0
Sep 17 00:51:48.548841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.548647    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"
Sep 17 00:51:48.548841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.548699    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" event={"ID":"1a97aaa1-df48-41b0-a7a3-830e8bbef608","Type":"ContainerDied","Data":"f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358"}
Sep 17 00:51:48.548841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.548764    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm" event={"ID":"1a97aaa1-df48-41b0-a7a3-830e8bbef608","Type":"ContainerDied","Data":"0082e5d0536376cd2d92a6685a2a1f5834f6db693a35d361e37277ec6bcab8ba"}
Sep 17 00:51:48.548841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.548796    2576 scope.go:117] "RemoveContainer" containerID="f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358"
Sep 17 00:51:48.559130 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.559112    2576 scope.go:117] "RemoveContainer" containerID="903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8"
Sep 17 00:51:48.568109 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.568086    2576 scope.go:117] "RemoveContainer" containerID="f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358"
Sep 17 00:51:48.568457 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:51:48.568435    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358\": container with ID starting with f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358 not found: ID does not exist" containerID="f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358"
Sep 17 00:51:48.568539 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.568465    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358"} err="failed to get container status \"f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358\": rpc error: code = NotFound desc = could not find container \"f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358\": container with ID starting with f0442b5e385ec53841d91b3af91dcda80a7d92144ba8e6ffd9d1014aed8a0358 not found: ID does not exist"
Sep 17 00:51:48.568539 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.568483    2576 scope.go:117] "RemoveContainer" containerID="903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8"
Sep 17 00:51:48.568834 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:51:48.568816    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8\": container with ID starting with 903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8 not found: ID does not exist" containerID="903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8"
Sep 17 00:51:48.568876 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.568841    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8"} err="failed to get container status \"903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8\": rpc error: code = NotFound desc = could not find container \"903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8\": container with ID starting with 903e44e3004fc3c3ececb38fdd65ba635e76dd7e8d40fc292e0b0ff8bf4614d8 not found: ID does not exist"
Sep 17 00:51:48.570201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.570180    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"]
Sep 17 00:51:48.573189 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.573167    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f40754-openshift-default-54d7dd9dfb-lp9gm"]
Sep 17 00:51:48.588619 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:48.588593    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" path="/var/lib/kubelet/pods/1a97aaa1-df48-41b0-a7a3-830e8bbef608/volumes"
Sep 17 00:51:49.674357 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:49.674324    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:49.777793 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:49.777769    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:49.876573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:49.876541    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:49.978429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:49.978350    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:50.571319 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:50.571257    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:50.670896 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:50.670868    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:50.774257 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:50.774221    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:52.149902 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:52.149871    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:54.670017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:54.669987    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:54.898644 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:54.898607    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:55.342228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:55.342191    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:55.616566 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:55.616476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:55.712110 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:55.712080    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:58.255398 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:58.255369    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:58.317260 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:58.317237    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:58.916190 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:58.916133    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:59.016948 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:59.016916    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:59.172226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:59.172128    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:59.269622 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:59.269588    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:51:59.720882 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:51:59.720852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:00.558766 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:00.558734    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:01.196750 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.196718    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:01.298281 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.298252    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:01.398333 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.398301    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:01.632023 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.631989    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:01.728429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.728396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:01.736507 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.736481    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"]
Sep 17 00:52:01.737081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.737058    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy"
Sep 17 00:52:01.737228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.737084    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy"
Sep 17 00:52:01.737228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.737115    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy"
Sep 17 00:52:01.737228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.737125    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy"
Sep 17 00:52:01.737228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.737165    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba12a105-636d-4d55-b5b7-b15ca90b5453" containerName="payload-processing"
Sep 17 00:52:01.737228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.737174    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba12a105-636d-4d55-b5b7-b15ca90b5453" containerName="payload-processing"
Sep 17 00:52:01.737501 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.737264    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy"
Sep 17 00:52:01.737501 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.737275    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba12a105-636d-4d55-b5b7-b15ca90b5453" containerName="payload-processing"
Sep 17 00:52:01.737501 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.737290    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a97aaa1-df48-41b0-a7a3-830e8bbef608" containerName="istio-proxy"
Sep 17 00:52:01.741997 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.741974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:01.744504 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.744480    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c98a88-dockercfg-gtswk\""
Sep 17 00:52:01.744648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.744623    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c98a88\""
Sep 17 00:52:01.751324 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.751256    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"]
Sep 17 00:52:01.775611 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.775582    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"]
Sep 17 00:52:01.779112 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.779096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:01.787070 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.787046    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"]
Sep 17 00:52:01.893027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.892937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wgjg\" (UniqueName: \"kubernetes.io/projected/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-kube-api-access-5wgjg\") pod \"payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk\" (UID: \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\") " pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:01.893027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.892991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2\" (UID: \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:01.893270 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.893061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2568\" (UniqueName: \"kubernetes.io/projected/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-kube-api-access-v2568\") pod \"payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2\" (UID: \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:01.893270 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.893122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk\" (UID: \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\") " pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:01.994283 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.994250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wgjg\" (UniqueName: \"kubernetes.io/projected/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-kube-api-access-5wgjg\") pod \"payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk\" (UID: \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\") " pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:01.994459 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.994298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2\" (UID: \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:01.994459 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.994344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2568\" (UniqueName: \"kubernetes.io/projected/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-kube-api-access-v2568\") pod \"payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2\" (UID: \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:01.994459 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.994392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk\" (UID: \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\") " pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:01.995314 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.995292    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk\" (UID: \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\") " pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:01.995370 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:01.995298    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2\" (UID: \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:02.002351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.002331    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wgjg\" (UniqueName: \"kubernetes.io/projected/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-kube-api-access-5wgjg\") pod \"payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk\" (UID: \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\") " pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:02.002467 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.002432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2568\" (UniqueName: \"kubernetes.io/projected/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-kube-api-access-v2568\") pod \"payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2\" (UID: \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:02.059412 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.059377    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:02.089489 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.089458    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:02.339346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.339316    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"]
Sep 17 00:52:02.341244 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:52:02.341216    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff8ff28_af84_4eb6_bcfd_6860ff05676b.slice/crio-e2d222e8c389a41aa61a72ab4417445d00758358ebef5659cb2fe2c0b5cc7d8c WatchSource:0}: Error finding container e2d222e8c389a41aa61a72ab4417445d00758358ebef5659cb2fe2c0b5cc7d8c: Status 404 returned error can't find the container with id e2d222e8c389a41aa61a72ab4417445d00758358ebef5659cb2fe2c0b5cc7d8c
Sep 17 00:52:02.378612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.378586    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"]
Sep 17 00:52:02.378939 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:52:02.378921    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5619d47_5256_4f2a_a9a7_f5ed0e5118ac.slice/crio-97fe7bc591f1d1c8777d17c8058d9ed1526cfc53d0622d9374cb079bf8979a00 WatchSource:0}: Error finding container 97fe7bc591f1d1c8777d17c8058d9ed1526cfc53d0622d9374cb079bf8979a00: Status 404 returned error can't find the container with id 97fe7bc591f1d1c8777d17c8058d9ed1526cfc53d0622d9374cb079bf8979a00
Sep 17 00:52:02.626581 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.626541    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk" event={"ID":"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac","Type":"ContainerStarted","Data":"9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2"}
Sep 17 00:52:02.626751 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.626603    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk" event={"ID":"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac","Type":"ContainerStarted","Data":"97fe7bc591f1d1c8777d17c8058d9ed1526cfc53d0622d9374cb079bf8979a00"}
Sep 17 00:52:02.626751 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.626660    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:02.628336 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.628312    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2" event={"ID":"5ff8ff28-af84-4eb6-bcfd-6860ff05676b","Type":"ContainerStarted","Data":"fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447"}
Sep 17 00:52:02.628434 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.628349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2" event={"ID":"5ff8ff28-af84-4eb6-bcfd-6860ff05676b","Type":"ContainerStarted","Data":"e2d222e8c389a41aa61a72ab4417445d00758358ebef5659cb2fe2c0b5cc7d8c"}
Sep 17 00:52:02.628434 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.628379    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:02.645116 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.645063    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk" podStartSLOduration=1.645043622 podStartE2EDuration="1.645043622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:52:02.641932033 +0000 UTC m=+1318.597528377" watchObservedRunningTime="2026-09-17 00:52:02.645043622 +0000 UTC m=+1318.600639966"
Sep 17 00:52:02.657986 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:02.657930    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2" podStartSLOduration=1.657914183 podStartE2EDuration="1.657914183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:52:02.655706493 +0000 UTC m=+1318.611302835" watchObservedRunningTime="2026-09-17 00:52:02.657914183 +0000 UTC m=+1318.613510529"
Sep 17 00:52:04.548493 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:04.548462    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:04.648639 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:04.648612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:04.790800 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:04.790762    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:05.065830 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:05.065801    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:05.273048 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:05.273012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:06.823290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:06.823254    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:07.608786 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:07.608752    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:10.173376 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:10.173346    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:10.813197 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:10.813165    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:10.907501 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:10.907464    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:11.236460 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:11.236375    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:11.959516 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:11.959478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:12.755370 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:12.755333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:13.558750 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:13.558723    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:13.635914 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:13.635890    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:52:13.636094 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:13.636012    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:52:14.472505 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:14.472449    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:14.572025 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:14.571997    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:14.803168 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:14.803123    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:14.920213 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:14.920115    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:15.016048 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:15.016019    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:15.473581 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:15.473552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:16.272485 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:16.272456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:17.056418 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:17.056382    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:17.687433 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:17.687394    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:18.440539 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:18.440503    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:18.941899 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:18.941854    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:19.076611 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:19.076581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:22.078226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:22.078184    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:26.981995 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:26.981919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:31.301465 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:31.301430    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:31.360554 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:31.360526    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:35.490184 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:35.490154    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:36.062227 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:36.062193    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:36.907354 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:36.907322    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:37.594978 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:37.594945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:38.382707 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:38.382677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:39.014335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:39.014298    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:39.869854 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:39.869820    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:40.529386 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:40.529351    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:40.870258 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:40.870174    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:41.283847 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:41.283818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:41.536640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:41.536554    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:41.635837 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:41.635809    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:41.956477 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:41.956399    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:47.771942 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:47.771915    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:47.871805 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:47.871779    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:48.114213 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.109320    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"]
Sep 17 00:52:48.118267 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.118223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"
Sep 17 00:52:48.119311 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.119282    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"]
Sep 17 00:52:48.120464 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.120443    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:52:48.196557 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.196535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f836f-0c43-4c31-becc-d6633cb7e2bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2\" (UID: \"052f836f-0c43-4c31-becc-d6633cb7e2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"
Sep 17 00:52:48.297540 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.297514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f836f-0c43-4c31-becc-d6633cb7e2bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2\" (UID: \"052f836f-0c43-4c31-becc-d6633cb7e2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"
Sep 17 00:52:48.298118 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.298101    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f836f-0c43-4c31-becc-d6633cb7e2bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2\" (UID: \"052f836f-0c43-4c31-becc-d6633cb7e2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"
Sep 17 00:52:48.431405 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.431345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"
Sep 17 00:52:48.708915 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.708894    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"]
Sep 17 00:52:48.710840 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:52:48.710811    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052f836f_0c43_4c31_becc_d6633cb7e2bb.slice/crio-4ab9f263c3ca1973a54d84c6c25d2e4957ffcac0dc63d8c5ae8bc2d938c3be57 WatchSource:0}: Error finding container 4ab9f263c3ca1973a54d84c6c25d2e4957ffcac0dc63d8c5ae8bc2d938c3be57: Status 404 returned error can't find the container with id 4ab9f263c3ca1973a54d84c6c25d2e4957ffcac0dc63d8c5ae8bc2d938c3be57
Sep 17 00:52:48.877043 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.877016    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:48.890290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:48.890267    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2" event={"ID":"052f836f-0c43-4c31-becc-d6633cb7e2bb","Type":"ContainerStarted","Data":"4ab9f263c3ca1973a54d84c6c25d2e4957ffcac0dc63d8c5ae8bc2d938c3be57"}
Sep 17 00:52:49.531332 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:49.531307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:49.897382 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:49.897299    2576 generic.go:356] "Generic (PLEG): container finished" podID="052f836f-0c43-4c31-becc-d6633cb7e2bb" containerID="5036c035968ad86b0d6d1d9a20828bfa67bc7388178da220599c0eb5fcaaf18c" exitCode=0
Sep 17 00:52:49.897720 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:49.897391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2" event={"ID":"052f836f-0c43-4c31-becc-d6633cb7e2bb","Type":"ContainerDied","Data":"5036c035968ad86b0d6d1d9a20828bfa67bc7388178da220599c0eb5fcaaf18c"}
Sep 17 00:52:50.165064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:50.164985    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:50.871915 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:50.871886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:51.052817 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:51.052796    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"
Sep 17 00:52:51.124002 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:51.123929    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f836f-0c43-4c31-becc-d6633cb7e2bb-maas-api-service-ca\") pod \"052f836f-0c43-4c31-becc-d6633cb7e2bb\" (UID: \"052f836f-0c43-4c31-becc-d6633cb7e2bb\") "
Sep 17 00:52:51.124281 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:51.124258    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052f836f-0c43-4c31-becc-d6633cb7e2bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "052f836f-0c43-4c31-becc-d6633cb7e2bb" (UID: "052f836f-0c43-4c31-becc-d6633cb7e2bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:52:51.225716 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:51.225690    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052f836f-0c43-4c31-becc-d6633cb7e2bb-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:52:51.472863 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:51.472788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:51.910653 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:51.910623    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"
Sep 17 00:52:51.910814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:51.910656    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2" event={"ID":"052f836f-0c43-4c31-becc-d6633cb7e2bb","Type":"ContainerDied","Data":"4ab9f263c3ca1973a54d84c6c25d2e4957ffcac0dc63d8c5ae8bc2d938c3be57"}
Sep 17 00:52:51.910814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:51.910700    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ab9f263c3ca1973a54d84c6c25d2e4957ffcac0dc63d8c5ae8bc2d938c3be57"
Sep 17 00:52:52.142259 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:52.142225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:58.539403 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:58.539331    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:58.600259 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:58.600225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:58.655672 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:58.655642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:58.713416 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:58.713390    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"]
Sep 17 00:52:58.715299 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:58.715274    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:58.715503 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:58.715484    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69d1f859-j8vb2"]
Sep 17 00:52:58.769098 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:58.769074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:59.089297 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:59.089257    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:59.189747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:59.189715    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:52:59.813131 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:52:59.813098    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:00.588371 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:00.588342    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052f836f-0c43-4c31-becc-d6633cb7e2bb" path="/var/lib/kubelet/pods/052f836f-0c43-4c31-becc-d6633cb7e2bb/volumes"
Sep 17 00:53:00.647040 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:00.647003    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:01.261415 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:01.261381    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:01.939711 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:01.939676    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:02.315819 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.315779    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:02.431222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.431189    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"]
Sep 17 00:53:02.431633 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.431621    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052f836f-0c43-4c31-becc-d6633cb7e2bb" containerName="revoke-keys"
Sep 17 00:53:02.431678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.431635    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="052f836f-0c43-4c31-becc-d6633cb7e2bb" containerName="revoke-keys"
Sep 17 00:53:02.431725 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.431717    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="052f836f-0c43-4c31-becc-d6633cb7e2bb" containerName="revoke-keys"
Sep 17 00:53:02.434971 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.434955    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"
Sep 17 00:53:02.437430 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.437404    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:53:02.442561 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.442536    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"]
Sep 17 00:53:02.517195 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.517163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7\" (UID: \"df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"
Sep 17 00:53:02.617857 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.617789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7\" (UID: \"df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"
Sep 17 00:53:02.618392 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.618373    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7\" (UID: \"df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"
Sep 17 00:53:02.747471 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.747440    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"
Sep 17 00:53:02.763297 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:02.763276    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:03.018550 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:03.018524    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"]
Sep 17 00:53:03.020789 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:53:03.020764    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf19a4d6_2b7f_494f_bd82_2c6c80c7b2ed.slice/crio-2faa7c36594aa8eb2ec1c96e9f0a4cb28d168fa6700a61c3481de2bd21a6dc05 WatchSource:0}: Error finding container 2faa7c36594aa8eb2ec1c96e9f0a4cb28d168fa6700a61c3481de2bd21a6dc05: Status 404 returned error can't find the container with id 2faa7c36594aa8eb2ec1c96e9f0a4cb28d168fa6700a61c3481de2bd21a6dc05
Sep 17 00:53:03.547255 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:03.547231    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:03.985501 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:03.985463    2576 generic.go:356] "Generic (PLEG): container finished" podID="df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed" containerID="e1121e6885e6ab7a9c386ed1947d0298b9784d3a3ca300bdd0e7750b5ac93052" exitCode=0
Sep 17 00:53:03.985652 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:03.985552    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7" event={"ID":"df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed","Type":"ContainerDied","Data":"e1121e6885e6ab7a9c386ed1947d0298b9784d3a3ca300bdd0e7750b5ac93052"}
Sep 17 00:53:03.985761 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:03.985651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7" event={"ID":"df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed","Type":"ContainerStarted","Data":"2faa7c36594aa8eb2ec1c96e9f0a4cb28d168fa6700a61c3481de2bd21a6dc05"}
Sep 17 00:53:04.266872 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:04.266788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:04.882803 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:04.882738    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:05.135540 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.135481    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"
Sep 17 00:53:05.241836 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.241806    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed-maas-api-service-ca\") pod \"df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed\" (UID: \"df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed\") "
Sep 17 00:53:05.242220 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.242193    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed" (UID: "df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:53:05.343286 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.343260    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:53:05.373982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.373956    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:05.474208 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.474125    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:05.627633 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.627606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:05.744809 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.744728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:05.951681 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.951649    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:05.998476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.998413    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"
Sep 17 00:53:05.998476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.998451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7" event={"ID":"df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed","Type":"ContainerDied","Data":"2faa7c36594aa8eb2ec1c96e9f0a4cb28d168fa6700a61c3481de2bd21a6dc05"}
Sep 17 00:53:05.998650 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:05.998494    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2faa7c36594aa8eb2ec1c96e9f0a4cb28d168fa6700a61c3481de2bd21a6dc05"
Sep 17 00:53:06.043158 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:06.043123    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:06.154116 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:06.154089    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:06.470120 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:06.470090    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:11.936190 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:11.936134    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:12.052892 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:12.052861    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:12.149408 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:12.149379    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:12.271933 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:12.271863    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:12.923891 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:12.923859    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:12.957265 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:12.957201    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"]
Sep 17 00:53:12.957921 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:12.957438    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk" podUID="e5619d47-5256-4f2a-a9a7-f5ed0e5118ac" containerName="payload-processing" containerID="cri-o://9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2" gracePeriod=30
Sep 17 00:53:12.965199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:12.965173    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"]
Sep 17 00:53:12.967083 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:12.965914    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2" podUID="5ff8ff28-af84-4eb6-bcfd-6860ff05676b" containerName="payload-pre-processing" containerID="cri-o://fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447" gracePeriod=30
Sep 17 00:53:12.998382 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:12.998361    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:13.057091 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.057052    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:13.108799 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.108763    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"]
Sep 17 00:53:13.112400 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.112375    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c98a88-8jkl7"]
Sep 17 00:53:13.113919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.113884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:13.249447 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.249429    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:53:13.262691 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.262666    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:53:13.416421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.416394    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-plugins-config-volume\") pod \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\" (UID: \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\") "
Sep 17 00:53:13.416615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.416514    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2568\" (UniqueName: \"kubernetes.io/projected/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-kube-api-access-v2568\") pod \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\" (UID: \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\") "
Sep 17 00:53:13.416615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.416543    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-plugins-config-volume\") pod \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\" (UID: \"5ff8ff28-af84-4eb6-bcfd-6860ff05676b\") "
Sep 17 00:53:13.416727 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.416613    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wgjg\" (UniqueName: \"kubernetes.io/projected/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-kube-api-access-5wgjg\") pod \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\" (UID: \"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac\") "
Sep 17 00:53:13.416917 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.416761    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5619d47-5256-4f2a-a9a7-f5ed0e5118ac" (UID: "e5619d47-5256-4f2a-a9a7-f5ed0e5118ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:53:13.416979 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.416910    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ff8ff28-af84-4eb6-bcfd-6860ff05676b" (UID: "5ff8ff28-af84-4eb6-bcfd-6860ff05676b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:53:13.418761 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.418736    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-kube-api-access-v2568" (OuterVolumeSpecName: "kube-api-access-v2568") pod "5ff8ff28-af84-4eb6-bcfd-6860ff05676b" (UID: "5ff8ff28-af84-4eb6-bcfd-6860ff05676b"). InnerVolumeSpecName "kube-api-access-v2568". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:53:13.418884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.418832    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-kube-api-access-5wgjg" (OuterVolumeSpecName: "kube-api-access-5wgjg") pod "e5619d47-5256-4f2a-a9a7-f5ed0e5118ac" (UID: "e5619d47-5256-4f2a-a9a7-f5ed0e5118ac"). InnerVolumeSpecName "kube-api-access-5wgjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:53:13.479258 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.479196    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:13.517635 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.517612    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wgjg\" (UniqueName: \"kubernetes.io/projected/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-kube-api-access-5wgjg\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:53:13.517635 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.517633    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:53:13.517773 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.517645    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2568\" (UniqueName: \"kubernetes.io/projected/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-kube-api-access-v2568\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:53:13.517773 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.517653    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff8ff28-af84-4eb6-bcfd-6860ff05676b-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:53:13.578770 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:13.578749    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:14.051087 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.051054    2576 generic.go:356] "Generic (PLEG): container finished" podID="e5619d47-5256-4f2a-a9a7-f5ed0e5118ac" containerID="9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2" exitCode=0
Sep 17 00:53:14.051475 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.051103    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"
Sep 17 00:53:14.051475 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.051137    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk" event={"ID":"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac","Type":"ContainerDied","Data":"9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2"}
Sep 17 00:53:14.051475 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.051225    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk" event={"ID":"e5619d47-5256-4f2a-a9a7-f5ed0e5118ac","Type":"ContainerDied","Data":"97fe7bc591f1d1c8777d17c8058d9ed1526cfc53d0622d9374cb079bf8979a00"}
Sep 17 00:53:14.051475 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.051258    2576 scope.go:117] "RemoveContainer" containerID="9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2"
Sep 17 00:53:14.052657 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.052632    2576 generic.go:356] "Generic (PLEG): container finished" podID="5ff8ff28-af84-4eb6-bcfd-6860ff05676b" containerID="fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447" exitCode=0
Sep 17 00:53:14.052792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.052671    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"
Sep 17 00:53:14.052792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.052725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2" event={"ID":"5ff8ff28-af84-4eb6-bcfd-6860ff05676b","Type":"ContainerDied","Data":"fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447"}
Sep 17 00:53:14.052792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.052765    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2" event={"ID":"5ff8ff28-af84-4eb6-bcfd-6860ff05676b","Type":"ContainerDied","Data":"e2d222e8c389a41aa61a72ab4417445d00758358ebef5659cb2fe2c0b5cc7d8c"}
Sep 17 00:53:14.060757 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.060739    2576 scope.go:117] "RemoveContainer" containerID="9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2"
Sep 17 00:53:14.061108 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:53:14.061086    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2\": container with ID starting with 9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2 not found: ID does not exist" containerID="9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2"
Sep 17 00:53:14.061232 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.061116    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2"} err="failed to get container status \"9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2\": rpc error: code = NotFound desc = could not find container \"9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2\": container with ID starting with 9dce8720a8e4613fb3c502c39901bfb4810e810b930ee8f4d8b2205c22998da2 not found: ID does not exist"
Sep 17 00:53:14.061232 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.061133    2576 scope.go:117] "RemoveContainer" containerID="fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447"
Sep 17 00:53:14.069524 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.069509    2576 scope.go:117] "RemoveContainer" containerID="fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447"
Sep 17 00:53:14.069760 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:53:14.069741    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447\": container with ID starting with fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447 not found: ID does not exist" containerID="fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447"
Sep 17 00:53:14.069821 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.069766    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447"} err="failed to get container status \"fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447\": rpc error: code = NotFound desc = could not find container \"fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447\": container with ID starting with fe96e61d07a1fc77522ae35cbab5ebbd5c2cda1812b0ae24d8251905f4dd5447 not found: ID does not exist"
Sep 17 00:53:14.075640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.075620    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"]
Sep 17 00:53:14.079471 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.079453    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c98a88-5bbbf5bcc-xgvnk"]
Sep 17 00:53:14.089065 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.089047    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"]
Sep 17 00:53:14.090936 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.090919    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c98a88-69b586bc58-npwn2"]
Sep 17 00:53:14.227691 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.227602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:14.590453 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.590424    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff8ff28-af84-4eb6-bcfd-6860ff05676b" path="/var/lib/kubelet/pods/5ff8ff28-af84-4eb6-bcfd-6860ff05676b/volumes"
Sep 17 00:53:14.590776 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.590763    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed" path="/var/lib/kubelet/pods/df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed/volumes"
Sep 17 00:53:14.591072 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.591061    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5619d47-5256-4f2a-a9a7-f5ed0e5118ac" path="/var/lib/kubelet/pods/e5619d47-5256-4f2a-a9a7-f5ed0e5118ac/volumes"
Sep 17 00:53:14.677601 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.677578    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"]
Sep 17 00:53:14.677982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.677962    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5619d47-5256-4f2a-a9a7-f5ed0e5118ac" containerName="payload-processing"
Sep 17 00:53:14.677982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.677981    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5619d47-5256-4f2a-a9a7-f5ed0e5118ac" containerName="payload-processing"
Sep 17 00:53:14.678421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.677997    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff8ff28-af84-4eb6-bcfd-6860ff05676b" containerName="payload-pre-processing"
Sep 17 00:53:14.678421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.678005    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff8ff28-af84-4eb6-bcfd-6860ff05676b" containerName="payload-pre-processing"
Sep 17 00:53:14.678421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.678039    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed" containerName="revoke-keys"
Sep 17 00:53:14.678421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.678048    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed" containerName="revoke-keys"
Sep 17 00:53:14.678421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.678185    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="df19a4d6-2b7f-494f-bd82-2c6c80c7b2ed" containerName="revoke-keys"
Sep 17 00:53:14.678421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.678203    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5619d47-5256-4f2a-a9a7-f5ed0e5118ac" containerName="payload-processing"
Sep 17 00:53:14.678421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.678214    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff8ff28-af84-4eb6-bcfd-6860ff05676b" containerName="payload-pre-processing"
Sep 17 00:53:14.682724 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.682707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.685042 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.685013    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"distinct-w0-a79f56-kserve-self-signed-certs\""
Sep 17 00:53:14.690536 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.690516    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"]
Sep 17 00:53:14.828421 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.828389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-dshm\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.828564 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.828453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-tmp-dir\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.828564 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.828501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpsz4\" (UniqueName: \"kubernetes.io/projected/2d2469d0-da52-435b-bc40-43a27fc06195-kube-api-access-bpsz4\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.828564 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.828524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2469d0-da52-435b-bc40-43a27fc06195-tls-certs\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.828564 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.828552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-home\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.828717 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.828569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-model-cache\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.887186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.887027    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:14.929675 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.929645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-dshm\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.929808 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.929704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-tmp-dir\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.929808 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.929738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpsz4\" (UniqueName: \"kubernetes.io/projected/2d2469d0-da52-435b-bc40-43a27fc06195-kube-api-access-bpsz4\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.929808 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.929759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2469d0-da52-435b-bc40-43a27fc06195-tls-certs\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.929808 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.929803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-home\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.930017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.929832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-model-cache\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.930133 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.930105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-tmp-dir\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.930218 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.930172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-model-cache\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.930283 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.930220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-home\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.932321 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.932293    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-dshm\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.932570 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.932553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2469d0-da52-435b-bc40-43a27fc06195-tls-certs\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.940453 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.940429    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpsz4\" (UniqueName: \"kubernetes.io/projected/2d2469d0-da52-435b-bc40-43a27fc06195-kube-api-access-bpsz4\") pod \"distinct-w0-a79f56-kserve-79f685ff9f-h2dj9\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") " pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:14.992841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:14.992809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:15.078009 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:15.077980    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:15.267874 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:15.267843    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"]
Sep 17 00:53:15.269630 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:53:15.269586    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d2469d0_da52_435b_bc40_43a27fc06195.slice/crio-8d50a26e0efe81f90b3ce7953b033d622d9cb2a7a2276b55703bb34e4c8b8240 WatchSource:0}: Error finding container 8d50a26e0efe81f90b3ce7953b033d622d9cb2a7a2276b55703bb34e4c8b8240: Status 404 returned error can't find the container with id 8d50a26e0efe81f90b3ce7953b033d622d9cb2a7a2276b55703bb34e4c8b8240
Sep 17 00:53:15.660436 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:15.660404    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:16.073201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:16.073168    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9" event={"ID":"2d2469d0-da52-435b-bc40-43a27fc06195","Type":"ContainerStarted","Data":"d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36"}
Sep 17 00:53:16.073378 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:16.073221    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9" event={"ID":"2d2469d0-da52-435b-bc40-43a27fc06195","Type":"ContainerStarted","Data":"8d50a26e0efe81f90b3ce7953b033d622d9cb2a7a2276b55703bb34e4c8b8240"}
Sep 17 00:53:16.089952 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:16.089908    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9" podStartSLOduration=2.089894875 podStartE2EDuration="2.089894875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:53:16.088288098 +0000 UTC m=+1392.043884439" watchObservedRunningTime="2026-09-17 00:53:16.089894875 +0000 UTC m=+1392.045491216"
Sep 17 00:53:17.777985 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:17.777948    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:17.932785 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:17.932755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:18.495285 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:18.495250    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:18.696288 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:18.696260    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:19.000469 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:19.000435    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:19.098938 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:19.098907    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:19.365133 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:19.365105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:19.420557 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:19.420531    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:19.605840 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:19.605812    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:20.043128 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.043092    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:20.302912 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.302829    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"]
Sep 17 00:53:20.306675 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.306650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.309014 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.308993    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-5a9565-openshift-default-dockercfg-mx4tm\""
Sep 17 00:53:20.320805 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.320778    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"]
Sep 17 00:53:20.333823 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.333795    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:20.374952 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.374925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-certs\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.375061 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.374966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-istio-token\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.375061 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.375020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-socket\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.375061 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.375035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-envoy\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.375061 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.375050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-data\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.375305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.375072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298105dd-1185-4edd-9a36-81c19c1011ad-istiod-ca-cert\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.375305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.375156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298105dd-1185-4edd-9a36-81c19c1011ad-istio-podinfo\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.375305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.375221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-credential-socket\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.375305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.375273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2qr\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-kube-api-access-zf2qr\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476341 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-istio-token\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476505 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-socket\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476505 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-envoy\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-data\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476621 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298105dd-1185-4edd-9a36-81c19c1011ad-istiod-ca-cert\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476731 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298105dd-1185-4edd-9a36-81c19c1011ad-istio-podinfo\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476731 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-credential-socket\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476830 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2qr\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-kube-api-access-zf2qr\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476881 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-socket\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476938 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-certs\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.476938 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.476894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-data\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.477103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.477078    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-credential-socket\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.477313 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.477093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-certs\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.477630 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.477610    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298105dd-1185-4edd-9a36-81c19c1011ad-istiod-ca-cert\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.478805 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.478787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-envoy\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.479178 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.479138    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298105dd-1185-4edd-9a36-81c19c1011ad-istio-podinfo\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.484137 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.484115    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-istio-token\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.484137 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.484130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2qr\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-kube-api-access-zf2qr\") pod \"e2e-noten-5a9565-openshift-default-7947d66586-r7299\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") " pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.524713 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.524694    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:20.633312 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.633244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:20.911590 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.911565    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"]
Sep 17 00:53:20.913244 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:53:20.913207    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298105dd_1185_4edd_9a36_81c19c1011ad.slice/crio-e3e55499cfafe413b424003be80b40f74debdbae7eda41a5c927ff5b7e48301e WatchSource:0}: Error finding container e3e55499cfafe413b424003be80b40f74debdbae7eda41a5c927ff5b7e48301e: Status 404 returned error can't find the container with id e3e55499cfafe413b424003be80b40f74debdbae7eda41a5c927ff5b7e48301e
Sep 17 00:53:20.915535 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.915501    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:53:20.915600 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.915590    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:53:20.915640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:20.915616    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:53:21.103580 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:21.103542    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299" event={"ID":"298105dd-1185-4edd-9a36-81c19c1011ad","Type":"ContainerStarted","Data":"397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b"}
Sep 17 00:53:21.103979 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:21.103604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299" event={"ID":"298105dd-1185-4edd-9a36-81c19c1011ad","Type":"ContainerStarted","Data":"e3e55499cfafe413b424003be80b40f74debdbae7eda41a5c927ff5b7e48301e"}
Sep 17 00:53:21.119932 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:21.119883    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299" podStartSLOduration=1.119864342 podStartE2EDuration="1.119864342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:53:21.119852574 +0000 UTC m=+1397.075448916" watchObservedRunningTime="2026-09-17 00:53:21.119864342 +0000 UTC m=+1397.075460686"
Sep 17 00:53:21.404499 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:21.404469    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:21.502851 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:21.502825    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:21.633726 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:21.633699    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:21.639785 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:21.639756    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:21.656512 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:21.656419    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:22.109447 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:22.109420    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:22.110363 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:22.110348    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:53:22.395884 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:22.395784    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:22.631594 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:22.631562    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:22.730137 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:22.730045    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:23.227966 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:23.227942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:23.948904 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:23.948870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:24.827449 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:24.827397    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:24.927451 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:24.927410    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:24.992924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:24.992897    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:24.993069 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:24.992960    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:25.005553 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:25.005531    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:25.028933 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:25.028909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:25.137566 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:25.137491    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:53:25.293304 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:25.293275    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:25.376401 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:25.376369    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:25.746711 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:25.746680    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:26.552356 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:26.552323    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:27.266080 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:27.266048    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:46.536829 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:46.536793    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:47.052350 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:47.052314    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:47.487945 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:47.487854    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:47.832731 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:47.832693    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:47.929944 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:47.929871    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:48.112933 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:48.112849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:48.308594 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:48.308564    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:48.666640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:48.666593    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:49.556084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:49.556052    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:50.451577 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:50.451531    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:50.690559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:50.690517    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:50.789862 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:50.789830    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:51.467294 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:51.467255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:52.351171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:52.351120    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:53.072131 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:53.072104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:53.741608 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:53.741550    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:54.461601 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:54.461572    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:54.518534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:54.518510    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:54.749889 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:54.749816    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:54.966070 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:54.966030    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:55.060627 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.060587    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:53:55.064965 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.064942    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"]
Sep 17 00:53:55.068875 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.068860    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:53:55.071268 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.071245    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-93017d\""
Sep 17 00:53:55.071374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.071296    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-93017d-dockercfg-st7kr\""
Sep 17 00:53:55.079028 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.079004    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"]
Sep 17 00:53:55.106261 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.106238    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"]
Sep 17 00:53:55.109733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.109717    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:53:55.116892 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.116874    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"]
Sep 17 00:53:55.180443 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.180409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33082d64-2f02-408e-b355-0d63dac4f89d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv\" (UID: \"33082d64-2f02-408e-b355-0d63dac4f89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:53:55.180881 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.180853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l5nx\" (UniqueName: \"kubernetes.io/projected/33082d64-2f02-408e-b355-0d63dac4f89d-kube-api-access-6l5nx\") pod \"payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv\" (UID: \"33082d64-2f02-408e-b355-0d63dac4f89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:53:55.181020 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.180900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjdb\" (UniqueName: \"kubernetes.io/projected/310eff3a-f4ac-472e-bc99-1833c4e885a0-kube-api-access-mzjdb\") pod \"payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m\" (UID: \"310eff3a-f4ac-472e-bc99-1833c4e885a0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:53:55.181020 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.180999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310eff3a-f4ac-472e-bc99-1833c4e885a0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m\" (UID: \"310eff3a-f4ac-472e-bc99-1833c4e885a0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:53:55.282162 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.282117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33082d64-2f02-408e-b355-0d63dac4f89d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv\" (UID: \"33082d64-2f02-408e-b355-0d63dac4f89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:53:55.282340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.282205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l5nx\" (UniqueName: \"kubernetes.io/projected/33082d64-2f02-408e-b355-0d63dac4f89d-kube-api-access-6l5nx\") pod \"payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv\" (UID: \"33082d64-2f02-408e-b355-0d63dac4f89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:53:55.282340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.282226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjdb\" (UniqueName: \"kubernetes.io/projected/310eff3a-f4ac-472e-bc99-1833c4e885a0-kube-api-access-mzjdb\") pod \"payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m\" (UID: \"310eff3a-f4ac-472e-bc99-1833c4e885a0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:53:55.282340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.282281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310eff3a-f4ac-472e-bc99-1833c4e885a0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m\" (UID: \"310eff3a-f4ac-472e-bc99-1833c4e885a0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:53:55.282878 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.282857    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33082d64-2f02-408e-b355-0d63dac4f89d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv\" (UID: \"33082d64-2f02-408e-b355-0d63dac4f89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:53:55.282974 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.282957    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310eff3a-f4ac-472e-bc99-1833c4e885a0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m\" (UID: \"310eff3a-f4ac-472e-bc99-1833c4e885a0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:53:55.290431 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.290403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjdb\" (UniqueName: \"kubernetes.io/projected/310eff3a-f4ac-472e-bc99-1833c4e885a0-kube-api-access-mzjdb\") pod \"payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m\" (UID: \"310eff3a-f4ac-472e-bc99-1833c4e885a0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:53:55.290577 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.290451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l5nx\" (UniqueName: \"kubernetes.io/projected/33082d64-2f02-408e-b355-0d63dac4f89d-kube-api-access-6l5nx\") pod \"payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv\" (UID: \"33082d64-2f02-408e-b355-0d63dac4f89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:53:55.380285 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.380230    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:53:55.420967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.420900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:53:55.659002 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.658942    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"]
Sep 17 00:53:55.661105 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:53:55.661078    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33082d64_2f02_408e_b355_0d63dac4f89d.slice/crio-2c82e8a43047e672aa43bc27061b8807a79449bf884231909847963f760e8132 WatchSource:0}: Error finding container 2c82e8a43047e672aa43bc27061b8807a79449bf884231909847963f760e8132: Status 404 returned error can't find the container with id 2c82e8a43047e672aa43bc27061b8807a79449bf884231909847963f760e8132
Sep 17 00:53:55.703977 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:55.703955    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"]
Sep 17 00:53:55.706063 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:53:55.706041    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310eff3a_f4ac_472e_bc99_1833c4e885a0.slice/crio-8d665f69d92ca5f04fb577095ebf37cf8c96e9c8cc18eb7d18b902dba912f6b5 WatchSource:0}: Error finding container 8d665f69d92ca5f04fb577095ebf37cf8c96e9c8cc18eb7d18b902dba912f6b5: Status 404 returned error can't find the container with id 8d665f69d92ca5f04fb577095ebf37cf8c96e9c8cc18eb7d18b902dba912f6b5
Sep 17 00:53:56.313040 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:56.313003    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m" event={"ID":"310eff3a-f4ac-472e-bc99-1833c4e885a0","Type":"ContainerStarted","Data":"e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd"}
Sep 17 00:53:56.313500 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:56.313072    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m" event={"ID":"310eff3a-f4ac-472e-bc99-1833c4e885a0","Type":"ContainerStarted","Data":"8d665f69d92ca5f04fb577095ebf37cf8c96e9c8cc18eb7d18b902dba912f6b5"}
Sep 17 00:53:56.313500 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:56.313095    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:53:56.314758 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:56.314733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv" event={"ID":"33082d64-2f02-408e-b355-0d63dac4f89d","Type":"ContainerStarted","Data":"5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a"}
Sep 17 00:53:56.314864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:56.314780    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv" event={"ID":"33082d64-2f02-408e-b355-0d63dac4f89d","Type":"ContainerStarted","Data":"2c82e8a43047e672aa43bc27061b8807a79449bf884231909847963f760e8132"}
Sep 17 00:53:56.314864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:56.314817    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:53:56.345678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:56.345630    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m" podStartSLOduration=1.34560969 podStartE2EDuration="1.34560969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:53:56.330185067 +0000 UTC m=+1432.285781408" watchObservedRunningTime="2026-09-17 00:53:56.34560969 +0000 UTC m=+1432.301206034"
Sep 17 00:53:56.347753 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:53:56.347713    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv" podStartSLOduration=1.347701325 podStartE2EDuration="1.347701325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:53:56.344916777 +0000 UTC m=+1432.300513118" watchObservedRunningTime="2026-09-17 00:53:56.347701325 +0000 UTC m=+1432.303297672"
Sep 17 00:54:07.323599 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:07.323558    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 00:54:07.323987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:07.323871    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 00:54:11.079269 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:11.079239    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:14.271374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:14.271339    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"]
Sep 17 00:54:14.293222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:14.293191    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:16.417613 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:16.417585    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:17.056470 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:17.056441    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:17.828201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:17.828138    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:18.561391 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:18.561356    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:19.244404 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:19.244366    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:20.047319 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:20.047268    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:20.722315 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:20.722281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:23.126725 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:23.126696    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:25.394770 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:25.394740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:25.496424 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:25.496391    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"]
Sep 17 00:54:25.496659 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:25.496634    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299" podUID="298105dd-1185-4edd-9a36-81c19c1011ad" containerName="istio-proxy" containerID="cri-o://397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b" gracePeriod=30
Sep 17 00:54:25.514686 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:25.514660    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:25.735961 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:25.735888    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:25.838732 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:25.838704    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:26.607481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:26.607448    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:26.696534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:26.696504    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"]
Sep 17 00:54:26.702186 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:26.702134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"
Sep 17 00:54:26.704681 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:26.704655    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:54:26.705580 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:26.705555    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"]
Sep 17 00:54:26.774631 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:26.774603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b7995-6465-4946-beb6-1fa603fb3f90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd\" (UID: \"1a1b7995-6465-4946-beb6-1fa603fb3f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"
Sep 17 00:54:26.876120 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:26.876051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b7995-6465-4946-beb6-1fa603fb3f90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd\" (UID: \"1a1b7995-6465-4946-beb6-1fa603fb3f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"
Sep 17 00:54:26.876639 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:26.876620    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b7995-6465-4946-beb6-1fa603fb3f90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd\" (UID: \"1a1b7995-6465-4946-beb6-1fa603fb3f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"
Sep 17 00:54:27.014881 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:27.014851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"
Sep 17 00:54:27.030595 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:27.030565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:27.287542 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:27.287516    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"]
Sep 17 00:54:27.289508 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:54:27.289473    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1b7995_6465_4946_beb6_1fa603fb3f90.slice/crio-a4edbf9b3968b4e00094c341a4d56aab0862e9c33b630566e5f0af2bac514548 WatchSource:0}: Error finding container a4edbf9b3968b4e00094c341a4d56aab0862e9c33b630566e5f0af2bac514548: Status 404 returned error can't find the container with id a4edbf9b3968b4e00094c341a4d56aab0862e9c33b630566e5f0af2bac514548
Sep 17 00:54:27.512289 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:27.512257    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd" event={"ID":"1a1b7995-6465-4946-beb6-1fa603fb3f90","Type":"ContainerStarted","Data":"a4edbf9b3968b4e00094c341a4d56aab0862e9c33b630566e5f0af2bac514548"}
Sep 17 00:54:27.742021 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:27.741989    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:28.464036 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:28.463963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:28.518926 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:28.518897    2576 generic.go:356] "Generic (PLEG): container finished" podID="1a1b7995-6465-4946-beb6-1fa603fb3f90" containerID="202c4e978c94ddc6096bb34668fe69d356f4b0a1fd53d9291715bc3a06120020" exitCode=0
Sep 17 00:54:28.519067 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:28.518963    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd" event={"ID":"1a1b7995-6465-4946-beb6-1fa603fb3f90","Type":"ContainerDied","Data":"202c4e978c94ddc6096bb34668fe69d356f4b0a1fd53d9291715bc3a06120020"}
Sep 17 00:54:29.155404 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.155365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:29.296632 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.296589    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb" podUID="43100e01-42bb-40c1-98f7-17c3ae215f0b" containerName="main" containerID="cri-o://1ad3b24a4368e9227bea1395a0362c48630709744ef80dd747aac87830c783f6" gracePeriod=45
Sep 17 00:54:29.526577 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.526545    2576 generic.go:356] "Generic (PLEG): container finished" podID="43100e01-42bb-40c1-98f7-17c3ae215f0b" containerID="1ad3b24a4368e9227bea1395a0362c48630709744ef80dd747aac87830c783f6" exitCode=0
Sep 17 00:54:29.526740 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.526618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb" event={"ID":"43100e01-42bb-40c1-98f7-17c3ae215f0b","Type":"ContainerDied","Data":"1ad3b24a4368e9227bea1395a0362c48630709744ef80dd747aac87830c783f6"}
Sep 17 00:54:29.564344 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.564319    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:54:29.694009 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.693986    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"
Sep 17 00:54:29.703327 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.703296    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-dshm\") pod \"43100e01-42bb-40c1-98f7-17c3ae215f0b\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") "
Sep 17 00:54:29.703437 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.703356    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-tmp-dir\") pod \"43100e01-42bb-40c1-98f7-17c3ae215f0b\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") "
Sep 17 00:54:29.703479 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.703451    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf8sj\" (UniqueName: \"kubernetes.io/projected/43100e01-42bb-40c1-98f7-17c3ae215f0b-kube-api-access-wf8sj\") pod \"43100e01-42bb-40c1-98f7-17c3ae215f0b\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") "
Sep 17 00:54:29.703515 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.703482    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43100e01-42bb-40c1-98f7-17c3ae215f0b-tls-certs\") pod \"43100e01-42bb-40c1-98f7-17c3ae215f0b\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") "
Sep 17 00:54:29.703566 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.703528    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-home\") pod \"43100e01-42bb-40c1-98f7-17c3ae215f0b\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") "
Sep 17 00:54:29.703624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.703561    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-model-cache\") pod \"43100e01-42bb-40c1-98f7-17c3ae215f0b\" (UID: \"43100e01-42bb-40c1-98f7-17c3ae215f0b\") "
Sep 17 00:54:29.703679 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.703656    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43100e01-42bb-40c1-98f7-17c3ae215f0b" (UID: "43100e01-42bb-40c1-98f7-17c3ae215f0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:29.703779 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.703757    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-home" (OuterVolumeSpecName: "home") pod "43100e01-42bb-40c1-98f7-17c3ae215f0b" (UID: "43100e01-42bb-40c1-98f7-17c3ae215f0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:29.703921 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.703901    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "43100e01-42bb-40c1-98f7-17c3ae215f0b" (UID: "43100e01-42bb-40c1-98f7-17c3ae215f0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:29.704182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.704082    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:29.704182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.704107    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:29.704182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.704123    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:29.706243 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.706214    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43100e01-42bb-40c1-98f7-17c3ae215f0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43100e01-42bb-40c1-98f7-17c3ae215f0b" (UID: "43100e01-42bb-40c1-98f7-17c3ae215f0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:54:29.706304 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.706246    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43100e01-42bb-40c1-98f7-17c3ae215f0b-kube-api-access-wf8sj" (OuterVolumeSpecName: "kube-api-access-wf8sj") pod "43100e01-42bb-40c1-98f7-17c3ae215f0b" (UID: "43100e01-42bb-40c1-98f7-17c3ae215f0b"). InnerVolumeSpecName "kube-api-access-wf8sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:54:29.706304 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.706249    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-dshm" (OuterVolumeSpecName: "dshm") pod "43100e01-42bb-40c1-98f7-17c3ae215f0b" (UID: "43100e01-42bb-40c1-98f7-17c3ae215f0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:29.805108 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.805087    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b7995-6465-4946-beb6-1fa603fb3f90-maas-api-service-ca\") pod \"1a1b7995-6465-4946-beb6-1fa603fb3f90\" (UID: \"1a1b7995-6465-4946-beb6-1fa603fb3f90\") "
Sep 17 00:54:29.805478 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.805453    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1b7995-6465-4946-beb6-1fa603fb3f90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a1b7995-6465-4946-beb6-1fa603fb3f90" (UID: "1a1b7995-6465-4946-beb6-1fa603fb3f90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:54:29.805532 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.805523    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8sj\" (UniqueName: \"kubernetes.io/projected/43100e01-42bb-40c1-98f7-17c3ae215f0b-kube-api-access-wf8sj\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:29.805568 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.805538    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43100e01-42bb-40c1-98f7-17c3ae215f0b-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:29.805568 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.805548    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43100e01-42bb-40c1-98f7-17c3ae215f0b-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:29.906490 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.906465    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b7995-6465-4946-beb6-1fa603fb3f90-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:29.956114 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:29.956041    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:30.057520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.057490    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:30.263341 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.263263    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"]
Sep 17 00:54:30.263909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.263894    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1b7995-6465-4946-beb6-1fa603fb3f90" containerName="revoke-keys"
Sep 17 00:54:30.263951 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.263912    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1b7995-6465-4946-beb6-1fa603fb3f90" containerName="revoke-keys"
Sep 17 00:54:30.263951 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.263928    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43100e01-42bb-40c1-98f7-17c3ae215f0b" containerName="main"
Sep 17 00:54:30.263951 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.263937    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43100e01-42bb-40c1-98f7-17c3ae215f0b" containerName="main"
Sep 17 00:54:30.264071 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.264059    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1b7995-6465-4946-beb6-1fa603fb3f90" containerName="revoke-keys"
Sep 17 00:54:30.264109 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.264078    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43100e01-42bb-40c1-98f7-17c3ae215f0b" containerName="main"
Sep 17 00:54:30.267665 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.267646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.269999 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.269971    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c7c845e1-openshift-default-dockercfg-mt4nv\""
Sep 17 00:54:30.279902 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.279878    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"]
Sep 17 00:54:30.289662 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.289638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:30.411665 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.411630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-socket\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.411821 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.411681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-podinfo\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.411821 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.411726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-certs\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.411821 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.411761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istiod-ca-cert\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.411952 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.411853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4x5b\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-kube-api-access-b4x5b\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.411952 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.411894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-token\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.412014 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.411949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-data\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.412014 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.411975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-envoy\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.412087 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.412015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-credential-socket\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513400 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-data\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513400 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-envoy\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513400 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-credential-socket\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-socket\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-podinfo\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-certs\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istiod-ca-cert\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4x5b\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-kube-api-access-b4x5b\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-token\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513701    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-data\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.513998 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.513965    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-certs\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.514174 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.514086    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-socket\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.514312 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.514282    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-credential-socket\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.514388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.514342    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istiod-ca-cert\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.516073 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.516053    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-envoy\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.516370 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.516354    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-podinfo\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.520722 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.520701    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-token\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.520935 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.520919    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4x5b\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-kube-api-access-b4x5b\") pod \"e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") " pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.534686 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.534661    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"
Sep 17 00:54:30.534784 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.534706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb" event={"ID":"43100e01-42bb-40c1-98f7-17c3ae215f0b","Type":"ContainerDied","Data":"6f79084e3c750a8a370a7c491038e810afa44f5bef3168cf7a8ee968220ace58"}
Sep 17 00:54:30.534784 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.534754    2576 scope.go:117] "RemoveContainer" containerID="1ad3b24a4368e9227bea1395a0362c48630709744ef80dd747aac87830c783f6"
Sep 17 00:54:30.536320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.536232    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd" event={"ID":"1a1b7995-6465-4946-beb6-1fa603fb3f90","Type":"ContainerDied","Data":"a4edbf9b3968b4e00094c341a4d56aab0862e9c33b630566e5f0af2bac514548"}
Sep 17 00:54:30.536320 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.536275    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"
Sep 17 00:54:30.536420 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.536280    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4edbf9b3968b4e00094c341a4d56aab0862e9c33b630566e5f0af2bac514548"
Sep 17 00:54:30.581610 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.581591    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:30.599067 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.599044    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"]
Sep 17 00:54:30.603485 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.603456    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e491c5/test-model-e491c5-kserve-7d949d77d7-vz6xb"]
Sep 17 00:54:30.629790 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.629710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:30.769216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.755616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:30.789046 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.789020    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:54:30.864825 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.864800    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:30.878444 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.878421    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"]
Sep 17 00:54:30.879210 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:54:30.879187    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ef85c3_30c3_4dde_b8d2_620370b4f53e.slice/crio-13ad0c93ee1f851c4c4216643e959340af5ac393a8486655c56c099cb1e77796 WatchSource:0}: Error finding container 13ad0c93ee1f851c4c4216643e959340af5ac393a8486655c56c099cb1e77796: Status 404 returned error can't find the container with id 13ad0c93ee1f851c4c4216643e959340af5ac393a8486655c56c099cb1e77796
Sep 17 00:54:30.881494 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.881466    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:54:30.881566 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.881527    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:54:30.881566 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.881554    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:54:30.917991 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.917969    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-socket\") pod \"298105dd-1185-4edd-9a36-81c19c1011ad\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") "
Sep 17 00:54:30.918084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918004    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-credential-socket\") pod \"298105dd-1185-4edd-9a36-81c19c1011ad\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") "
Sep 17 00:54:30.918084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918060    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf2qr\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-kube-api-access-zf2qr\") pod \"298105dd-1185-4edd-9a36-81c19c1011ad\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") "
Sep 17 00:54:30.918236 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918087    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298105dd-1185-4edd-9a36-81c19c1011ad-istiod-ca-cert\") pod \"298105dd-1185-4edd-9a36-81c19c1011ad\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") "
Sep 17 00:54:30.918236 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918107    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-envoy\") pod \"298105dd-1185-4edd-9a36-81c19c1011ad\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") "
Sep 17 00:54:30.918236 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918135    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-data\") pod \"298105dd-1185-4edd-9a36-81c19c1011ad\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") "
Sep 17 00:54:30.918236 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918184    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-certs\") pod \"298105dd-1185-4edd-9a36-81c19c1011ad\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") "
Sep 17 00:54:30.918236 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918209    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-istio-token\") pod \"298105dd-1185-4edd-9a36-81c19c1011ad\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") "
Sep 17 00:54:30.918492 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918249    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "298105dd-1185-4edd-9a36-81c19c1011ad" (UID: "298105dd-1185-4edd-9a36-81c19c1011ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:30.918492 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918305    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "298105dd-1185-4edd-9a36-81c19c1011ad" (UID: "298105dd-1185-4edd-9a36-81c19c1011ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:30.918492 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918315    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298105dd-1185-4edd-9a36-81c19c1011ad-istio-podinfo\") pod \"298105dd-1185-4edd-9a36-81c19c1011ad\" (UID: \"298105dd-1185-4edd-9a36-81c19c1011ad\") "
Sep 17 00:54:30.918492 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918458    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "298105dd-1185-4edd-9a36-81c19c1011ad" (UID: "298105dd-1185-4edd-9a36-81c19c1011ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:30.918492 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918470    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "298105dd-1185-4edd-9a36-81c19c1011ad" (UID: "298105dd-1185-4edd-9a36-81c19c1011ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:30.918769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918647    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298105dd-1185-4edd-9a36-81c19c1011ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "298105dd-1185-4edd-9a36-81c19c1011ad" (UID: "298105dd-1185-4edd-9a36-81c19c1011ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:54:30.918769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918661    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:30.918769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918682    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:30.918769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918694    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:30.918769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.918707    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:30.920770 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.920743    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "298105dd-1185-4edd-9a36-81c19c1011ad" (UID: "298105dd-1185-4edd-9a36-81c19c1011ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:30.920921 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.920883    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-kube-api-access-zf2qr" (OuterVolumeSpecName: "kube-api-access-zf2qr") pod "298105dd-1185-4edd-9a36-81c19c1011ad" (UID: "298105dd-1185-4edd-9a36-81c19c1011ad"). InnerVolumeSpecName "kube-api-access-zf2qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:54:30.921023 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.921009    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "298105dd-1185-4edd-9a36-81c19c1011ad" (UID: "298105dd-1185-4edd-9a36-81c19c1011ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:54:30.921122 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:30.921103    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/298105dd-1185-4edd-9a36-81c19c1011ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "298105dd-1185-4edd-9a36-81c19c1011ad" (UID: "298105dd-1185-4edd-9a36-81c19c1011ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:54:31.019902 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.019810    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298105dd-1185-4edd-9a36-81c19c1011ad-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:31.019902 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.019842    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf2qr\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-kube-api-access-zf2qr\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:31.019902 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.019857    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298105dd-1185-4edd-9a36-81c19c1011ad-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:31.019902 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.019870    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298105dd-1185-4edd-9a36-81c19c1011ad-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:31.019902 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.019881    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298105dd-1185-4edd-9a36-81c19c1011ad-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:31.443663 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.443629    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:31.546129 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.546089    2576 generic.go:356] "Generic (PLEG): container finished" podID="298105dd-1185-4edd-9a36-81c19c1011ad" containerID="397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b" exitCode=0
Sep 17 00:54:31.546346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.546167    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"
Sep 17 00:54:31.546346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.546184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299" event={"ID":"298105dd-1185-4edd-9a36-81c19c1011ad","Type":"ContainerDied","Data":"397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b"}
Sep 17 00:54:31.546346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.546241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299" event={"ID":"298105dd-1185-4edd-9a36-81c19c1011ad","Type":"ContainerDied","Data":"e3e55499cfafe413b424003be80b40f74debdbae7eda41a5c927ff5b7e48301e"}
Sep 17 00:54:31.546346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.546271    2576 scope.go:117] "RemoveContainer" containerID="397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b"
Sep 17 00:54:31.549524 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.549497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" event={"ID":"21ef85c3-30c3-4dde-b8d2-620370b4f53e","Type":"ContainerStarted","Data":"f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6"}
Sep 17 00:54:31.549632 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.549546    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" event={"ID":"21ef85c3-30c3-4dde-b8d2-620370b4f53e","Type":"ContainerStarted","Data":"13ad0c93ee1f851c4c4216643e959340af5ac393a8486655c56c099cb1e77796"}
Sep 17 00:54:31.556656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.556639    2576 scope.go:117] "RemoveContainer" containerID="397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b"
Sep 17 00:54:31.556909 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:54:31.556889    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b\": container with ID starting with 397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b not found: ID does not exist" containerID="397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b"
Sep 17 00:54:31.556970 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.556914    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b"} err="failed to get container status \"397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b\": rpc error: code = NotFound desc = could not find container \"397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b\": container with ID starting with 397cd8f43a5f0a012872f2dbd0cc3f30ca60c39aa9673f05ba7c0b7a1e56c75b not found: ID does not exist"
Sep 17 00:54:31.567640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.567575    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" podStartSLOduration=1.5675616749999999 podStartE2EDuration="1.567561675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:31.566748131 +0000 UTC m=+1467.522344473" watchObservedRunningTime="2026-09-17 00:54:31.567561675 +0000 UTC m=+1467.523158015"
Sep 17 00:54:31.580841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.580821    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"]
Sep 17 00:54:31.581798 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.581781    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:31.584511 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.584491    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-5a9565-openshift-default-7947d66586-r7299"]
Sep 17 00:54:31.587102 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:31.587088    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:32.556111 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:32.556074    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:32.557065 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:32.557047    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:32.589500 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:32.589471    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="298105dd-1185-4edd-9a36-81c19c1011ad" path="/var/lib/kubelet/pods/298105dd-1185-4edd-9a36-81c19c1011ad/volumes"
Sep 17 00:54:32.590081 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:32.590060    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43100e01-42bb-40c1-98f7-17c3ae215f0b" path="/var/lib/kubelet/pods/43100e01-42bb-40c1-98f7-17c3ae215f0b/volumes"
Sep 17 00:54:32.668612 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:32.668571    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:33.391605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:33.391572    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:33.499261 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:33.499209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:33.592625 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:33.592600    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:34.192806 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:34.192778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:35.066427 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:35.066389    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:35.782569 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:35.782537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:36.402252 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:36.402212    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:37.025127 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:37.025099    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:37.659608 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:37.659579    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:37.715873 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:37.715847    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:38.014094 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.014016    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:38.115498 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.115463    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:38.206864 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.206837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:38.214548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.214522    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"]
Sep 17 00:54:38.214974 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.214960    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298105dd-1185-4edd-9a36-81c19c1011ad" containerName="istio-proxy"
Sep 17 00:54:38.215017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.214977    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="298105dd-1185-4edd-9a36-81c19c1011ad" containerName="istio-proxy"
Sep 17 00:54:38.215083 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.215072    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="298105dd-1185-4edd-9a36-81c19c1011ad" containerName="istio-proxy"
Sep 17 00:54:38.218388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.218374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:38.221434 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.221412    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c7c845e1-dockercfg-9qbjx\""
Sep 17 00:54:38.221560 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.221412    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c7c845e1\""
Sep 17 00:54:38.228982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.228963    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"]
Sep 17 00:54:38.247767 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.247743    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"]
Sep 17 00:54:38.251271 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.251254    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:38.260252 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.260228    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"]
Sep 17 00:54:38.390304 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.390269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r54m\" (UniqueName: \"kubernetes.io/projected/eb8f9dbf-03a8-449a-87d9-aa0082042f52-kube-api-access-5r54m\") pod \"payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq\" (UID: \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:38.390481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.390397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/8e6cbb2e-006d-4515-9707-c11828945c08-kube-api-access-h8ffd\") pod \"payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n\" (UID: \"8e6cbb2e-006d-4515-9707-c11828945c08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:38.390481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.390465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6cbb2e-006d-4515-9707-c11828945c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n\" (UID: \"8e6cbb2e-006d-4515-9707-c11828945c08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:38.390557 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.390499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f9dbf-03a8-449a-87d9-aa0082042f52-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq\" (UID: \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:38.491165 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.491114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6cbb2e-006d-4515-9707-c11828945c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n\" (UID: \"8e6cbb2e-006d-4515-9707-c11828945c08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:38.491305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.491171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f9dbf-03a8-449a-87d9-aa0082042f52-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq\" (UID: \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:38.491305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.491208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r54m\" (UniqueName: \"kubernetes.io/projected/eb8f9dbf-03a8-449a-87d9-aa0082042f52-kube-api-access-5r54m\") pod \"payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq\" (UID: \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:38.491378 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.491306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/8e6cbb2e-006d-4515-9707-c11828945c08-kube-api-access-h8ffd\") pod \"payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n\" (UID: \"8e6cbb2e-006d-4515-9707-c11828945c08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:38.491947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.491926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f9dbf-03a8-449a-87d9-aa0082042f52-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq\" (UID: \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:38.491987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.491929    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6cbb2e-006d-4515-9707-c11828945c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n\" (UID: \"8e6cbb2e-006d-4515-9707-c11828945c08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:38.499990 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.499971    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/8e6cbb2e-006d-4515-9707-c11828945c08-kube-api-access-h8ffd\") pod \"payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n\" (UID: \"8e6cbb2e-006d-4515-9707-c11828945c08\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:38.500064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.500016    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r54m\" (UniqueName: \"kubernetes.io/projected/eb8f9dbf-03a8-449a-87d9-aa0082042f52-kube-api-access-5r54m\") pod \"payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq\" (UID: \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\") " pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:38.528853 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.528788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:38.563014 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.562757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:38.567084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.567051    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:38.812435 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.812413    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"]
Sep 17 00:54:38.813635 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:54:38.813606    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6cbb2e_006d_4515_9707_c11828945c08.slice/crio-f1fc46af445607f26c3aa0fd1631749701fd56bd4c35c7a778b88a2ca27bf629 WatchSource:0}: Error finding container f1fc46af445607f26c3aa0fd1631749701fd56bd4c35c7a778b88a2ca27bf629: Status 404 returned error can't find the container with id f1fc46af445607f26c3aa0fd1631749701fd56bd4c35c7a778b88a2ca27bf629
Sep 17 00:54:38.852833 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:38.852808    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"]
Sep 17 00:54:38.859236 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:54:38.859211    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8f9dbf_03a8_449a_87d9_aa0082042f52.slice/crio-dfa89419e6fa0bacbff9d0ee7231f1aa42fedeecfe703d8d77697fd7fc5829fa WatchSource:0}: Error finding container dfa89419e6fa0bacbff9d0ee7231f1aa42fedeecfe703d8d77697fd7fc5829fa: Status 404 returned error can't find the container with id dfa89419e6fa0bacbff9d0ee7231f1aa42fedeecfe703d8d77697fd7fc5829fa
Sep 17 00:54:39.147082 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.146998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:39.174005 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.173972    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"]
Sep 17 00:54:39.174304 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.174259    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6" podUID="98ad4838-8ed1-4a6b-b202-9964ca6e3c7f" containerName="payload-processing" containerID="cri-o://70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483" gracePeriod=30
Sep 17 00:54:39.244459 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.244425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:39.323907 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.323515    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"]
Sep 17 00:54:39.324942 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.324918    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:39.326773 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.326752    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e491c5-hgjqd"]
Sep 17 00:54:39.382470 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.382436    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:39.430407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.430387    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:54:39.455930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.454587    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:39.603949 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.603910    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fhk5\" (UniqueName: \"kubernetes.io/projected/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-kube-api-access-2fhk5\") pod \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\" (UID: \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\") "
Sep 17 00:54:39.603949 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.603943    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-plugins-config-volume\") pod \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\" (UID: \"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f\") "
Sep 17 00:54:39.604588 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.604555    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98ad4838-8ed1-4a6b-b202-9964ca6e3c7f" (UID: "98ad4838-8ed1-4a6b-b202-9964ca6e3c7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:54:39.606828 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.606795    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-kube-api-access-2fhk5" (OuterVolumeSpecName: "kube-api-access-2fhk5") pod "98ad4838-8ed1-4a6b-b202-9964ca6e3c7f" (UID: "98ad4838-8ed1-4a6b-b202-9964ca6e3c7f"). InnerVolumeSpecName "kube-api-access-2fhk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:54:39.607235 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.607211    2576 generic.go:356] "Generic (PLEG): container finished" podID="98ad4838-8ed1-4a6b-b202-9964ca6e3c7f" containerID="70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483" exitCode=0
Sep 17 00:54:39.607335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.607269    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"
Sep 17 00:54:39.607335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.607291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6" event={"ID":"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f","Type":"ContainerDied","Data":"70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483"}
Sep 17 00:54:39.607452 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.607358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6" event={"ID":"98ad4838-8ed1-4a6b-b202-9964ca6e3c7f","Type":"ContainerDied","Data":"d93c23d83fc36af9c3db239ab1b62ba1ed8bec7edf52402ef4081a41cdd56782"}
Sep 17 00:54:39.607452 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.607389    2576 scope.go:117] "RemoveContainer" containerID="70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483"
Sep 17 00:54:39.610308 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.609860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n" event={"ID":"8e6cbb2e-006d-4515-9707-c11828945c08","Type":"ContainerStarted","Data":"264b980f59b34904e7b2741c54c7931ca96d78acc29666d64cc13b7ccc30f302"}
Sep 17 00:54:39.610308 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.609909    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n" event={"ID":"8e6cbb2e-006d-4515-9707-c11828945c08","Type":"ContainerStarted","Data":"f1fc46af445607f26c3aa0fd1631749701fd56bd4c35c7a778b88a2ca27bf629"}
Sep 17 00:54:39.610308 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.609937    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:39.612603 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.612533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq" event={"ID":"eb8f9dbf-03a8-449a-87d9-aa0082042f52","Type":"ContainerStarted","Data":"e231761d3a067de2697a99f40a200e7c72e6ca4f3abb4fb8d378f1c3d19b5192"}
Sep 17 00:54:39.612603 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.612579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq" event={"ID":"eb8f9dbf-03a8-449a-87d9-aa0082042f52","Type":"ContainerStarted","Data":"dfa89419e6fa0bacbff9d0ee7231f1aa42fedeecfe703d8d77697fd7fc5829fa"}
Sep 17 00:54:39.612759 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.612609    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:39.624520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.624505    2576 scope.go:117] "RemoveContainer" containerID="70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483"
Sep 17 00:54:39.624781 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:54:39.624763    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483\": container with ID starting with 70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483 not found: ID does not exist" containerID="70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483"
Sep 17 00:54:39.624840 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.624789    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483"} err="failed to get container status \"70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483\": rpc error: code = NotFound desc = could not find container \"70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483\": container with ID starting with 70ad40e989c914b7f0607d0230caefe3ba976ab1b28f0ce2d2fa5417d040a483 not found: ID does not exist"
Sep 17 00:54:39.627009 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.626972    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n" podStartSLOduration=1.626958851 podStartE2EDuration="1.626958851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:39.62546924 +0000 UTC m=+1475.581065582" watchObservedRunningTime="2026-09-17 00:54:39.626958851 +0000 UTC m=+1475.582555221"
Sep 17 00:54:39.640615 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.640585    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"]
Sep 17 00:54:39.644858 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.644832    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e491c5-85c7659467-68kx6"]
Sep 17 00:54:39.658985 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.658945    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq" podStartSLOduration=1.658932063 podStartE2EDuration="1.658932063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:39.657647632 +0000 UTC m=+1475.613243973" watchObservedRunningTime="2026-09-17 00:54:39.658932063 +0000 UTC m=+1475.614528404"
Sep 17 00:54:39.671770 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.671748    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:39.705210 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.705122    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fhk5\" (UniqueName: \"kubernetes.io/projected/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-kube-api-access-2fhk5\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:39.705210 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:39.705180    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:40.076124 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:40.076086    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:40.588990 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:40.588962    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1b7995-6465-4946-beb6-1fa603fb3f90" path="/var/lib/kubelet/pods/1a1b7995-6465-4946-beb6-1fa603fb3f90/volumes"
Sep 17 00:54:40.589348 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:40.589335    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ad4838-8ed1-4a6b-b202-9964ca6e3c7f" path="/var/lib/kubelet/pods/98ad4838-8ed1-4a6b-b202-9964ca6e3c7f/volumes"
Sep 17 00:54:40.701478 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:40.701449    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:41.461064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:41.461037    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:42.138109 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:42.138068    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:42.770338 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:42.770307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:43.654288 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:43.654237    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:45.306754 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.306721    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:45.365787 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.365757    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:45.559483 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.559407    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"]
Sep 17 00:54:45.559660 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.559645    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy" containerID="cri-o://556d812375e3bf8ed0f086d0969f62288b337012adccf01578f8f94974ac1eb6" gracePeriod=30
Sep 17 00:54:45.577539 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.577461    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:45.916193 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.916085    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"]
Sep 17 00:54:45.916534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.916521    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ad4838-8ed1-4a6b-b202-9964ca6e3c7f" containerName="payload-processing"
Sep 17 00:54:45.916586 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.916536    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ad4838-8ed1-4a6b-b202-9964ca6e3c7f" containerName="payload-processing"
Sep 17 00:54:45.916631 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.916621    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ad4838-8ed1-4a6b-b202-9964ca6e3c7f" containerName="payload-processing"
Sep 17 00:54:45.921118 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.921101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"
Sep 17 00:54:45.923545 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.923520    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:54:45.925818 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:45.925787    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"]
Sep 17 00:54:46.067851 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:46.067813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ba5edf-a66e-4246-b566-8335fee33aa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9\" (UID: \"57ba5edf-a66e-4246-b566-8335fee33aa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"
Sep 17 00:54:46.169331 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:46.169259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ba5edf-a66e-4246-b566-8335fee33aa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9\" (UID: \"57ba5edf-a66e-4246-b566-8335fee33aa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"
Sep 17 00:54:46.169989 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:46.169964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ba5edf-a66e-4246-b566-8335fee33aa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9\" (UID: \"57ba5edf-a66e-4246-b566-8335fee33aa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"
Sep 17 00:54:46.215379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:46.215355    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:46.233408 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:46.233374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"
Sep 17 00:54:46.506843 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:46.506819    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"]
Sep 17 00:54:46.508676 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:54:46.508650    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ba5edf_a66e_4246_b566_8335fee33aa9.slice/crio-60d7ac05836e93d17646e0f30eb849556e0ffea5cb9faee9763b68d16eb7916d WatchSource:0}: Error finding container 60d7ac05836e93d17646e0f30eb849556e0ffea5cb9faee9763b68d16eb7916d: Status 404 returned error can't find the container with id 60d7ac05836e93d17646e0f30eb849556e0ffea5cb9faee9763b68d16eb7916d
Sep 17 00:54:46.655799 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:46.655770    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9" event={"ID":"57ba5edf-a66e-4246-b566-8335fee33aa9","Type":"ContainerStarted","Data":"60d7ac05836e93d17646e0f30eb849556e0ffea5cb9faee9763b68d16eb7916d"}
Sep 17 00:54:46.841133 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:46.841106    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:47.567199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:47.567172    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:47.669585 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:47.669548    2576 generic.go:356] "Generic (PLEG): container finished" podID="57ba5edf-a66e-4246-b566-8335fee33aa9" containerID="c6c216afdab0e84ba2bcbdb3966d055677b3a80a01e10cab1420a2adaa52443e" exitCode=0
Sep 17 00:54:47.669749 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:47.669610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9" event={"ID":"57ba5edf-a66e-4246-b566-8335fee33aa9","Type":"ContainerDied","Data":"c6c216afdab0e84ba2bcbdb3966d055677b3a80a01e10cab1420a2adaa52443e"}
Sep 17 00:54:48.242527 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:48.242498    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:48.844769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:48.844459    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"
Sep 17 00:54:48.997882 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:48.997797    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ba5edf-a66e-4246-b566-8335fee33aa9-maas-api-service-ca\") pod \"57ba5edf-a66e-4246-b566-8335fee33aa9\" (UID: \"57ba5edf-a66e-4246-b566-8335fee33aa9\") "
Sep 17 00:54:48.998128 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:48.998106    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ba5edf-a66e-4246-b566-8335fee33aa9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57ba5edf-a66e-4246-b566-8335fee33aa9" (UID: "57ba5edf-a66e-4246-b566-8335fee33aa9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:54:49.099027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:49.098993    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ba5edf-a66e-4246-b566-8335fee33aa9-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:49.141944 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:49.141907    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:49.684729 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:49.684689    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9" event={"ID":"57ba5edf-a66e-4246-b566-8335fee33aa9","Type":"ContainerDied","Data":"60d7ac05836e93d17646e0f30eb849556e0ffea5cb9faee9763b68d16eb7916d"}
Sep 17 00:54:49.684897 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:49.684742    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60d7ac05836e93d17646e0f30eb849556e0ffea5cb9faee9763b68d16eb7916d"
Sep 17 00:54:49.684897 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:49.684784    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"
Sep 17 00:54:49.770670 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:49.770643    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:50.551712 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.551656    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:50.622389 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.622364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:54:50.622537 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.622446    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:54:50.697724 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.697685    2576 generic.go:356] "Generic (PLEG): container finished" podID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerID="556d812375e3bf8ed0f086d0969f62288b337012adccf01578f8f94974ac1eb6" exitCode=0
Sep 17 00:54:50.697921 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.697742    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" event={"ID":"161c2fcd-59bb-46b4-8899-45cbac6e01ff","Type":"ContainerDied","Data":"556d812375e3bf8ed0f086d0969f62288b337012adccf01578f8f94974ac1eb6"}
Sep 17 00:54:50.697921 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.697802    2576 scope.go:117] "RemoveContainer" containerID="f4f655c65eb369b86078520804ac7a787ff1e5c71651d7540f2ed4070c2ef1cd"
Sep 17 00:54:50.869632 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.869604    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:54:50.916657 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.916625    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istiod-ca-cert\") pod \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") "
Sep 17 00:54:50.916827 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.916666    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg8pd\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-kube-api-access-sg8pd\") pod \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") "
Sep 17 00:54:50.916827 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.916729    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-envoy\") pod \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") "
Sep 17 00:54:50.916827 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.916790    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-socket\") pod \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") "
Sep 17 00:54:50.916827 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.916814    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-podinfo\") pod \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") "
Sep 17 00:54:50.917038 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.916835    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-certs\") pod \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") "
Sep 17 00:54:50.917038 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.916868    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-token\") pod \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") "
Sep 17 00:54:50.917038 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.916903    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "161c2fcd-59bb-46b4-8899-45cbac6e01ff" (UID: "161c2fcd-59bb-46b4-8899-45cbac6e01ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:54:50.917038 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.916910    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-data\") pod \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") "
Sep 17 00:54:50.917258 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917094    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "161c2fcd-59bb-46b4-8899-45cbac6e01ff" (UID: "161c2fcd-59bb-46b4-8899-45cbac6e01ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:50.917258 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917004    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-credential-socket\") pod \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\" (UID: \"161c2fcd-59bb-46b4-8899-45cbac6e01ff\") "
Sep 17 00:54:50.917258 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917166    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "161c2fcd-59bb-46b4-8899-45cbac6e01ff" (UID: "161c2fcd-59bb-46b4-8899-45cbac6e01ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:50.917390 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917254    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "161c2fcd-59bb-46b4-8899-45cbac6e01ff" (UID: "161c2fcd-59bb-46b4-8899-45cbac6e01ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:50.917449 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917429    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "161c2fcd-59bb-46b4-8899-45cbac6e01ff" (UID: "161c2fcd-59bb-46b4-8899-45cbac6e01ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:50.917638 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917618    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:50.917638 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917639    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:50.917824 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917651    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:50.917824 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917663    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:50.917824 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.917673    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:50.919559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.919536    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "161c2fcd-59bb-46b4-8899-45cbac6e01ff" (UID: "161c2fcd-59bb-46b4-8899-45cbac6e01ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:54:50.919640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.919557    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-kube-api-access-sg8pd" (OuterVolumeSpecName: "kube-api-access-sg8pd") pod "161c2fcd-59bb-46b4-8899-45cbac6e01ff" (UID: "161c2fcd-59bb-46b4-8899-45cbac6e01ff"). InnerVolumeSpecName "kube-api-access-sg8pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:54:50.919790 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.919775    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "161c2fcd-59bb-46b4-8899-45cbac6e01ff" (UID: "161c2fcd-59bb-46b4-8899-45cbac6e01ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:54:50.919900 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:50.919886    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "161c2fcd-59bb-46b4-8899-45cbac6e01ff" (UID: "161c2fcd-59bb-46b4-8899-45cbac6e01ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:54:51.018586 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.018552    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:51.018586 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.018582    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg8pd\" (UniqueName: \"kubernetes.io/projected/161c2fcd-59bb-46b4-8899-45cbac6e01ff-kube-api-access-sg8pd\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:51.018586 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.018594    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:51.018793 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.018604    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c2fcd-59bb-46b4-8899-45cbac6e01ff-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:51.167585 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.167491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:51.705426 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.705400    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"
Sep 17 00:54:51.705879 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.705396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs" event={"ID":"161c2fcd-59bb-46b4-8899-45cbac6e01ff","Type":"ContainerDied","Data":"33bf67f118056b0485c0c50bac3a23faec3cc08a8a2cd4143add64f4dcda752a"}
Sep 17 00:54:51.705879 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.705537    2576 scope.go:117] "RemoveContainer" containerID="556d812375e3bf8ed0f086d0969f62288b337012adccf01578f8f94974ac1eb6"
Sep 17 00:54:51.726872 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.726849    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"]
Sep 17 00:54:51.729968 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.729942    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e491c5-openshift-default-f477c6b8d-z4zgs"]
Sep 17 00:54:51.785538 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:51.785513    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:52.549999 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:52.549954    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:52.588456 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:52.588425    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" path="/var/lib/kubelet/pods/161c2fcd-59bb-46b4-8899-45cbac6e01ff/volumes"
Sep 17 00:54:53.206386 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:53.206356    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:53.306481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:53.306449    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:56.045413 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.045369    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:56.587343 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.587308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:56.620780 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.620747    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"]
Sep 17 00:54:56.621017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.620989    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl" podUID="dffca83f-2117-43b0-9846-295c6cda4a5a" containerName="payload-pre-processing" containerID="cri-o://a7dcf206d04d4d76e1ef8ae5dc893bf3cf91a1531d6316db43e491cae6f91373" gracePeriod=30
Sep 17 00:54:56.684574 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.684549    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:56.739748 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.739676    2576 generic.go:356] "Generic (PLEG): container finished" podID="dffca83f-2117-43b0-9846-295c6cda4a5a" containerID="a7dcf206d04d4d76e1ef8ae5dc893bf3cf91a1531d6316db43e491cae6f91373" exitCode=0
Sep 17 00:54:56.739915 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.739779    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl" event={"ID":"dffca83f-2117-43b0-9846-295c6cda4a5a","Type":"ContainerDied","Data":"a7dcf206d04d4d76e1ef8ae5dc893bf3cf91a1531d6316db43e491cae6f91373"}
Sep 17 00:54:56.769481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.769448    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"]
Sep 17 00:54:56.773365 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.773304    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e421ba-82gs9"]
Sep 17 00:54:56.773728 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.773708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:56.823984 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.823939    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:56.881771 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.881745    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:54:56.978042 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.978017    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jz84\" (UniqueName: \"kubernetes.io/projected/dffca83f-2117-43b0-9846-295c6cda4a5a-kube-api-access-4jz84\") pod \"dffca83f-2117-43b0-9846-295c6cda4a5a\" (UID: \"dffca83f-2117-43b0-9846-295c6cda4a5a\") "
Sep 17 00:54:56.978213 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.978165    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffca83f-2117-43b0-9846-295c6cda4a5a-plugins-config-volume\") pod \"dffca83f-2117-43b0-9846-295c6cda4a5a\" (UID: \"dffca83f-2117-43b0-9846-295c6cda4a5a\") "
Sep 17 00:54:56.978532 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.978508    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffca83f-2117-43b0-9846-295c6cda4a5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dffca83f-2117-43b0-9846-295c6cda4a5a" (UID: "dffca83f-2117-43b0-9846-295c6cda4a5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:54:56.980222 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:56.980200    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffca83f-2117-43b0-9846-295c6cda4a5a-kube-api-access-4jz84" (OuterVolumeSpecName: "kube-api-access-4jz84") pod "dffca83f-2117-43b0-9846-295c6cda4a5a" (UID: "dffca83f-2117-43b0-9846-295c6cda4a5a"). InnerVolumeSpecName "kube-api-access-4jz84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:54:57.079165 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.079126    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jz84\" (UniqueName: \"kubernetes.io/projected/dffca83f-2117-43b0-9846-295c6cda4a5a-kube-api-access-4jz84\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:57.079165 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.079166    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dffca83f-2117-43b0-9846-295c6cda4a5a-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:54:57.113138 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.113113    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:57.213686 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.213663    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:57.747173 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.747126    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"
Sep 17 00:54:57.747347 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.747173    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl" event={"ID":"dffca83f-2117-43b0-9846-295c6cda4a5a","Type":"ContainerDied","Data":"59ebc3ba0f709fc00074344a40b86ccdc5f5c5f37ea059fbf8c685a4d093c880"}
Sep 17 00:54:57.747347 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.747243    2576 scope.go:117] "RemoveContainer" containerID="a7dcf206d04d4d76e1ef8ae5dc893bf3cf91a1531d6316db43e491cae6f91373"
Sep 17 00:54:57.749200 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.749170    2576 generic.go:356] "Generic (PLEG): container finished" podID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerID="f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6" exitCode=0
Sep 17 00:54:57.749336 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.749233    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" event={"ID":"21ef85c3-30c3-4dde-b8d2-620370b4f53e","Type":"ContainerDied","Data":"f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6"}
Sep 17 00:54:57.749643 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.749627    2576 scope.go:117] "RemoveContainer" containerID="f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6"
Sep 17 00:54:57.750768 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.750740    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:54:57.750893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.750814    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:54:57.750893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.750849    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:54:57.802793 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.802668    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"]
Sep 17 00:54:57.804945 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.804916    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e421ba-58cfcdd6bb-4srjl"]
Sep 17 00:54:57.862909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:57.862877    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:58.588534 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:58.588497    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ba5edf-a66e-4246-b566-8335fee33aa9" path="/var/lib/kubelet/pods/57ba5edf-a66e-4246-b566-8335fee33aa9/volumes"
Sep 17 00:54:58.588984 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:58.588967    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dffca83f-2117-43b0-9846-295c6cda4a5a" path="/var/lib/kubelet/pods/dffca83f-2117-43b0-9846-295c6cda4a5a/volumes"
Sep 17 00:54:58.649188 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:58.649129    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:58.762249 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:58.762212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" event={"ID":"21ef85c3-30c3-4dde-b8d2-620370b4f53e","Type":"ContainerStarted","Data":"b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483"}
Sep 17 00:54:59.280160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:59.280116    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:54:59.581888 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:59.581801    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:54:59.583189 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:59.583159    2576 patch_prober.go:28] interesting pod/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 00:54:59.583336 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:54:59.583212    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 00:55:00.056194 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:00.056161    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:00.582109 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:00.582070    2576 patch_prober.go:28] interesting pod/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 00:55:00.582302 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:00.582160    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 00:55:01.581966 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:01.581935    2576 patch_prober.go:28] interesting pod/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 00:55:01.582461 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:01.581989    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 00:55:02.556335 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:02.556295    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:55:02.582814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:02.582779    2576 patch_prober.go:28] interesting pod/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 00:55:02.583209 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:02.582848    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 00:55:02.821728 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:02.821646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:02.883282 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:02.883233    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:03.175351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.175258    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"]
Sep 17 00:55:03.175541 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.175515    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy" containerID="cri-o://1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5" gracePeriod=30
Sep 17 00:55:03.202230 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.202203    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:03.582065 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.582022    2576 patch_prober.go:28] interesting pod/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 00:55:03.582351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.582103    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 00:55:03.684025 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.683989    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:03.967559 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.967479    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"]
Sep 17 00:55:03.968160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968041    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffca83f-2117-43b0-9846-295c6cda4a5a" containerName="payload-pre-processing"
Sep 17 00:55:03.968160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968067    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffca83f-2117-43b0-9846-295c6cda4a5a" containerName="payload-pre-processing"
Sep 17 00:55:03.968160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968092    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy"
Sep 17 00:55:03.968160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968099    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy"
Sep 17 00:55:03.968160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968160    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ba5edf-a66e-4246-b566-8335fee33aa9" containerName="revoke-keys"
Sep 17 00:55:03.968160 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968170    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ba5edf-a66e-4246-b566-8335fee33aa9" containerName="revoke-keys"
Sep 17 00:55:03.968548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968284    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy"
Sep 17 00:55:03.968548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968297    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy"
Sep 17 00:55:03.968548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968308    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dffca83f-2117-43b0-9846-295c6cda4a5a" containerName="payload-pre-processing"
Sep 17 00:55:03.968548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.968319    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ba5edf-a66e-4246-b566-8335fee33aa9" containerName="revoke-keys"
Sep 17 00:55:03.971530 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.971496    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:03.974203 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.973984    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d2326d-openshift-default-dockercfg-zjv85\""
Sep 17 00:55:03.983617 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.983596    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"]
Sep 17 00:55:03.993672 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:03.993641    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:04.063872 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.063840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-istio-token\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.064032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.063902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556fc\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-kube-api-access-556fc\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.064032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.063950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-envoy\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.064032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.064013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f50785f-463a-4c64-9447-c635eb575d94-istio-podinfo\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.064261 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.064098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-data\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.064261 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.064159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-credential-socket\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.064261 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.064195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f50785f-463a-4c64-9447-c635eb575d94-istiod-ca-cert\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.064261 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.064237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-certs\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.064476 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.064281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-socket\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.164992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-socket\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-istio-token\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556fc\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-kube-api-access-556fc\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-envoy\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f50785f-463a-4c64-9447-c635eb575d94-istio-podinfo\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-data\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165542 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-credential-socket\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165542 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f50785f-463a-4c64-9447-c635eb575d94-istiod-ca-cert\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165542 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-certs\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165542 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165474    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-socket\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.165881 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-credential-socket\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.166001 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-certs\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.166001 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.165925    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-data\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.166181 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.166133    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f50785f-463a-4c64-9447-c635eb575d94-istiod-ca-cert\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.168414 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.168385    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-envoy\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.168520 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.168505    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f50785f-463a-4c64-9447-c635eb575d94-istio-podinfo\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.173597 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.173567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-556fc\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-kube-api-access-556fc\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.173736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.173716    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-istio-token\") pod \"e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") " pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.198526 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.198497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:04.286153 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.286119    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:04.560629 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.560605    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"]
Sep 17 00:55:04.562486 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:55:04.562455    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f50785f_463a_4c64_9447_c635eb575d94.slice/crio-a181965ee4e5eb9c8d0a81690d05f12364700966e68c2ee7da6a148c5d9cab21 WatchSource:0}: Error finding container a181965ee4e5eb9c8d0a81690d05f12364700966e68c2ee7da6a148c5d9cab21: Status 404 returned error can't find the container with id a181965ee4e5eb9c8d0a81690d05f12364700966e68c2ee7da6a148c5d9cab21
Sep 17 00:55:04.564819 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.564776    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:55:04.564957 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.564851    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:55:04.564957 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.564894    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:55:04.590062 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.590043    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:55:04.591232 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.591214    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:55:04.688223 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.688198    2576 scope.go:117] "RemoveContainer" containerID="6409055046c734c55502f5ee15a3bd2bce415f81f09257e9bc34dcb053565a61"
Sep 17 00:55:04.699908 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.699883    2576 scope.go:117] "RemoveContainer" containerID="a7f4c07e65ca72fb70fcba41f103d8d5fe575e4019ae3917e0843c4e22fdd0ee"
Sep 17 00:55:04.802308 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.802274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" event={"ID":"5f50785f-463a-4c64-9447-c635eb575d94","Type":"ContainerStarted","Data":"cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93"}
Sep 17 00:55:04.802440 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.802326    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" event={"ID":"5f50785f-463a-4c64-9447-c635eb575d94","Type":"ContainerStarted","Data":"a181965ee4e5eb9c8d0a81690d05f12364700966e68c2ee7da6a148c5d9cab21"}
Sep 17 00:55:04.825297 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:04.825210    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" podStartSLOduration=1.825193335 podStartE2EDuration="1.825193335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:55:04.82367938 +0000 UTC m=+1500.779275734" watchObservedRunningTime="2026-09-17 00:55:04.825193335 +0000 UTC m=+1500.780789676"
Sep 17 00:55:05.052401 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:05.052374    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:55:05.053721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:05.053698    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 00:55:05.286766 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:05.286737    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:05.288454 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:05.288426    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 00:55:05.288569 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:05.288485    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 00:55:06.057881 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:06.057844    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:06.291316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:06.291289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:06.671202 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:06.671130    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:06.774617 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:06.774573    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:06.813299 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:06.813274    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:06.814333 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:06.814317    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:06.837423 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:06.837390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:07.568489 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:07.568461    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:08.111954 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.111918    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:08.432215 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.432187    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:55:08.505370 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505342    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-credential-socket\") pod \"e8cb7738-42b7-4cb1-8b01-65b65835510f\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") "
Sep 17 00:55:08.505518 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505411    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-envoy\") pod \"e8cb7738-42b7-4cb1-8b01-65b65835510f\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") "
Sep 17 00:55:08.505518 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505457    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-socket\") pod \"e8cb7738-42b7-4cb1-8b01-65b65835510f\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") "
Sep 17 00:55:08.505518 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505478    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb7738-42b7-4cb1-8b01-65b65835510f-istiod-ca-cert\") pod \"e8cb7738-42b7-4cb1-8b01-65b65835510f\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") "
Sep 17 00:55:08.505518 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505503    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-certs\") pod \"e8cb7738-42b7-4cb1-8b01-65b65835510f\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") "
Sep 17 00:55:08.505742 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505529    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7vtf\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-kube-api-access-f7vtf\") pod \"e8cb7738-42b7-4cb1-8b01-65b65835510f\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") "
Sep 17 00:55:08.505742 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505560    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8cb7738-42b7-4cb1-8b01-65b65835510f" (UID: "e8cb7738-42b7-4cb1-8b01-65b65835510f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:55:08.505742 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505604    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-data\") pod \"e8cb7738-42b7-4cb1-8b01-65b65835510f\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") "
Sep 17 00:55:08.505742 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505629    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-podinfo\") pod \"e8cb7738-42b7-4cb1-8b01-65b65835510f\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") "
Sep 17 00:55:08.505742 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505679    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-token\") pod \"e8cb7738-42b7-4cb1-8b01-65b65835510f\" (UID: \"e8cb7738-42b7-4cb1-8b01-65b65835510f\") "
Sep 17 00:55:08.506024 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505753    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8cb7738-42b7-4cb1-8b01-65b65835510f" (UID: "e8cb7738-42b7-4cb1-8b01-65b65835510f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:55:08.506024 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.505825    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8cb7738-42b7-4cb1-8b01-65b65835510f" (UID: "e8cb7738-42b7-4cb1-8b01-65b65835510f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:55:08.506316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.506031    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8cb7738-42b7-4cb1-8b01-65b65835510f" (UID: "e8cb7738-42b7-4cb1-8b01-65b65835510f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:55:08.506316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.506052    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cb7738-42b7-4cb1-8b01-65b65835510f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8cb7738-42b7-4cb1-8b01-65b65835510f" (UID: "e8cb7738-42b7-4cb1-8b01-65b65835510f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:55:08.506316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.506171    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:55:08.506316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.506193    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb7738-42b7-4cb1-8b01-65b65835510f-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:55:08.506316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.506208    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:55:08.506316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.506221    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:55:08.506316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.506235    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:55:08.507943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.507919    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8cb7738-42b7-4cb1-8b01-65b65835510f" (UID: "e8cb7738-42b7-4cb1-8b01-65b65835510f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:55:08.508042 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.507942    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8cb7738-42b7-4cb1-8b01-65b65835510f" (UID: "e8cb7738-42b7-4cb1-8b01-65b65835510f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:55:08.508042 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.508009    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-kube-api-access-f7vtf" (OuterVolumeSpecName: "kube-api-access-f7vtf") pod "e8cb7738-42b7-4cb1-8b01-65b65835510f" (UID: "e8cb7738-42b7-4cb1-8b01-65b65835510f"). InnerVolumeSpecName "kube-api-access-f7vtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:55:08.508587 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.508564    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8cb7738-42b7-4cb1-8b01-65b65835510f" (UID: "e8cb7738-42b7-4cb1-8b01-65b65835510f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:55:08.607402 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.607383    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:55:08.607678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.607404    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7vtf\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-kube-api-access-f7vtf\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:55:08.607678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.607415    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:55:08.607678 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.607423    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cb7738-42b7-4cb1-8b01-65b65835510f-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:55:08.825981 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.825943    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerID="1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5" exitCode=0
Sep 17 00:55:08.826165 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.825994    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"
Sep 17 00:55:08.826165 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.826029    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" event={"ID":"e8cb7738-42b7-4cb1-8b01-65b65835510f","Type":"ContainerDied","Data":"1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5"}
Sep 17 00:55:08.826165 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.826120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk" event={"ID":"e8cb7738-42b7-4cb1-8b01-65b65835510f","Type":"ContainerDied","Data":"2e389ca63eb4dde8b2c833b82102bc5de3e6569602b5e77e55488114ac28796c"}
Sep 17 00:55:08.826290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.826181    2576 scope.go:117] "RemoveContainer" containerID="1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5"
Sep 17 00:55:08.835449 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.835432    2576 scope.go:117] "RemoveContainer" containerID="ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184"
Sep 17 00:55:08.844809 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.844790    2576 scope.go:117] "RemoveContainer" containerID="1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5"
Sep 17 00:55:08.845163 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:55:08.845119    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5\": container with ID starting with 1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5 not found: ID does not exist" containerID="1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5"
Sep 17 00:55:08.845271 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.845177    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5"} err="failed to get container status \"1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5\": rpc error: code = NotFound desc = could not find container \"1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5\": container with ID starting with 1b054736929ec734627c43b27da1f5e3d0b1af08a721597fbd8fce8c63a4d2a5 not found: ID does not exist"
Sep 17 00:55:08.845271 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.845203    2576 scope.go:117] "RemoveContainer" containerID="ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184"
Sep 17 00:55:08.845734 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:55:08.845524    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184\": container with ID starting with ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184 not found: ID does not exist" containerID="ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184"
Sep 17 00:55:08.845734 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.845541    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"]
Sep 17 00:55:08.845734 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.845551    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184"} err="failed to get container status \"ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184\": rpc error: code = NotFound desc = could not find container \"ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184\": container with ID starting with ccba64e9c01b4eed16c0fbaa4f7e0519bc4cc26ca5a524996c766bcc6b5c2184 not found: ID does not exist"
Sep 17 00:55:08.848127 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:08.848108    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e421ba-openshift-default-685765cb4b-d9dqk"]
Sep 17 00:55:09.266838 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:09.266758    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:09.791763 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:09.791735    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:09.849025 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:09.848991    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:09.968064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:09.968034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:10.236239 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.236129    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:10.334937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.334911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:10.373560 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.373520    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"]
Sep 17 00:55:10.374011 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.373993    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy"
Sep 17 00:55:10.374107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.374015    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy"
Sep 17 00:55:10.374107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.374054    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy"
Sep 17 00:55:10.374107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.374064    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="161c2fcd-59bb-46b4-8899-45cbac6e01ff" containerName="istio-proxy"
Sep 17 00:55:10.374306 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.374204    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy"
Sep 17 00:55:10.378869 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.378847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:10.382519 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.382497    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d2326d\""
Sep 17 00:55:10.382643 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.382563    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d2326d-dockercfg-7th5g\""
Sep 17 00:55:10.384908 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.384885    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"]
Sep 17 00:55:10.524842 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.524752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj\" (UID: \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:10.525006 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.524906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drv2\" (UniqueName: \"kubernetes.io/projected/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-kube-api-access-9drv2\") pod \"payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj\" (UID: \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:10.588226 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.588198    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" path="/var/lib/kubelet/pods/e8cb7738-42b7-4cb1-8b01-65b65835510f/volumes"
Sep 17 00:55:10.626323 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.626295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9drv2\" (UniqueName: \"kubernetes.io/projected/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-kube-api-access-9drv2\") pod \"payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj\" (UID: \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:10.626456 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.626361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj\" (UID: \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:10.626976 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.626959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj\" (UID: \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:10.634085 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.634057    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drv2\" (UniqueName: \"kubernetes.io/projected/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-kube-api-access-9drv2\") pod \"payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj\" (UID: \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:10.692548 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.692528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:10.969407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:10.969382    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"]
Sep 17 00:55:10.970664 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:55:10.970639    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95a5e91_97b0_4ac8_86e8_d40f8fde0175.slice/crio-92e1802f8d592d7675c518aeaa5469322946e1fd564fcb3186563eb38abb7647 WatchSource:0}: Error finding container 92e1802f8d592d7675c518aeaa5469322946e1fd564fcb3186563eb38abb7647: Status 404 returned error can't find the container with id 92e1802f8d592d7675c518aeaa5469322946e1fd564fcb3186563eb38abb7647
Sep 17 00:55:11.848648 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:11.848610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj" event={"ID":"a95a5e91-97b0-4ac8-86e8-d40f8fde0175","Type":"ContainerStarted","Data":"7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea"}
Sep 17 00:55:11.848845 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:11.848673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj" event={"ID":"a95a5e91-97b0-4ac8-86e8-d40f8fde0175","Type":"ContainerStarted","Data":"92e1802f8d592d7675c518aeaa5469322946e1fd564fcb3186563eb38abb7647"}
Sep 17 00:55:11.848845 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:11.848708    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:11.865420 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:11.865378    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj" podStartSLOduration=1.865365364 podStartE2EDuration="1.865365364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:55:11.86324795 +0000 UTC m=+1507.818844292" watchObservedRunningTime="2026-09-17 00:55:11.865365364 +0000 UTC m=+1507.820961705"
Sep 17 00:55:20.688100 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:20.688067    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:22.856729 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:22.856698    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 00:55:31.633397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:31.633342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:31.739169 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:31.736231    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:32.090088 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:32.090051    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:32.390275 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:32.390137    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:32.486478 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:32.486449    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:33.062253 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:33.062221    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:33.162728 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:33.162700    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:33.826374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:33.826339    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:34.004096 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:34.004067    2576 generic.go:356] "Generic (PLEG): container finished" podID="5f50785f-463a-4c64-9447-c635eb575d94" containerID="cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93" exitCode=0
Sep 17 00:55:34.004275 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:34.004161    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" event={"ID":"5f50785f-463a-4c64-9447-c635eb575d94","Type":"ContainerDied","Data":"cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93"}
Sep 17 00:55:34.004554 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:34.004534    2576 scope.go:117] "RemoveContainer" containerID="cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93"
Sep 17 00:55:34.005737 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:34.005715    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:55:34.005789 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:34.005775    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:55:34.005825 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:34.005807    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:55:34.615027 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:34.614986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:35.011344 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:35.011259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" event={"ID":"5f50785f-463a-4c64-9447-c635eb575d94","Type":"ContainerStarted","Data":"c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b"}
Sep 17 00:55:35.231111 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:35.231077    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:35.286441 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:35.286412    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:35.287896 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:35.287868    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 00:55:35.287994 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:35.287931    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 00:55:35.334067 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:35.334038    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:35.432528 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:35.432477    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:36.257723 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:36.257612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:36.286908 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:36.286882    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 00:55:36.287029 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:36.286941    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 00:55:36.813652 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:36.813622    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:36.966668 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:36.966606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:37.286909 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:37.286881    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 00:55:37.287297 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:37.286946    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 00:55:37.586953 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:37.586868    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:38.287118 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:38.287081    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 00:55:38.287613 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:38.287186    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 00:55:38.357168 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:38.357120    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:38.881333 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:38.881294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:38.980979 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:38.980944    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:39.286747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:39.286715    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 00:55:39.286910 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:39.286781    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 00:55:39.744242 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:39.744095    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:39.844751 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:39.844708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:39.942172 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:39.942115    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:40.190166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.190078    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:40.291167 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.291114    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:40.292327 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.292304    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 00:55:40.298393 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.298365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:40.300661 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.300637    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"]
Sep 17 00:55:40.301047 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.301034    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy"
Sep 17 00:55:40.301095 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.301049    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy"
Sep 17 00:55:40.301159 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.301133    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8cb7738-42b7-4cb1-8b01-65b65835510f" containerName="istio-proxy"
Sep 17 00:55:40.304471 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.304452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:55:40.306690 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.306670    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bcd112-dockercfg-2dwbd\""
Sep 17 00:55:40.306791 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.306709    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bcd112\""
Sep 17 00:55:40.315055 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.314997    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"]
Sep 17 00:55:40.418557 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.418522    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb085f8-5546-4b70-9293-1b3e437745b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf\" (UID: \"2fb085f8-5546-4b70-9293-1b3e437745b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:55:40.418769 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.418738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86z5\" (UniqueName: \"kubernetes.io/projected/2fb085f8-5546-4b70-9293-1b3e437745b3-kube-api-access-j86z5\") pod \"payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf\" (UID: \"2fb085f8-5546-4b70-9293-1b3e437745b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:55:40.519455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.519348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb085f8-5546-4b70-9293-1b3e437745b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf\" (UID: \"2fb085f8-5546-4b70-9293-1b3e437745b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:55:40.519455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.519442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86z5\" (UniqueName: \"kubernetes.io/projected/2fb085f8-5546-4b70-9293-1b3e437745b3-kube-api-access-j86z5\") pod \"payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf\" (UID: \"2fb085f8-5546-4b70-9293-1b3e437745b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:55:40.520121 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.520102    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb085f8-5546-4b70-9293-1b3e437745b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf\" (UID: \"2fb085f8-5546-4b70-9293-1b3e437745b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:55:40.527251 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.527226    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86z5\" (UniqueName: \"kubernetes.io/projected/2fb085f8-5546-4b70-9293-1b3e437745b3-kube-api-access-j86z5\") pod \"payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf\" (UID: \"2fb085f8-5546-4b70-9293-1b3e437745b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:55:40.617311 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.617283    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:55:40.891025 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:40.891002    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"]
Sep 17 00:55:40.893335 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:55:40.893304    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb085f8_5546_4b70_9293_1b3e437745b3.slice/crio-b97483c69ebdaf954cf377eabb341deda4dc6ec51d679316fed645c6cc1c6c1a WatchSource:0}: Error finding container b97483c69ebdaf954cf377eabb341deda4dc6ec51d679316fed645c6cc1c6c1a: Status 404 returned error can't find the container with id b97483c69ebdaf954cf377eabb341deda4dc6ec51d679316fed645c6cc1c6c1a
Sep 17 00:55:41.048029 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:41.047992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf" event={"ID":"2fb085f8-5546-4b70-9293-1b3e437745b3","Type":"ContainerStarted","Data":"f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541"}
Sep 17 00:55:41.048248 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:41.048057    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf" event={"ID":"2fb085f8-5546-4b70-9293-1b3e437745b3","Type":"ContainerStarted","Data":"b97483c69ebdaf954cf377eabb341deda4dc6ec51d679316fed645c6cc1c6c1a"}
Sep 17 00:55:41.048248 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:41.048085    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:55:41.069773 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:41.069730    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf" podStartSLOduration=1.069714631 podStartE2EDuration="1.069714631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:55:41.067890207 +0000 UTC m=+1537.023486548" watchObservedRunningTime="2026-09-17 00:55:41.069714631 +0000 UTC m=+1537.025310973"
Sep 17 00:55:41.896907 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:41.896877    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:41.998454 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:41.998427    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:55:52.056092 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:55:52.056063    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 00:56:01.767279 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:01.767246    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:02.547064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:02.547029    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:03.447763 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:03.447710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:04.154579 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:04.154545    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:04.754913 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:04.754881    2576 scope.go:117] "RemoveContainer" containerID="068b66ccc11c78805ec0e35023d710466115de59df20c6187cd714fd627c294f"
Sep 17 00:56:04.850578 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:04.850551    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:05.400849 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:05.400820    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:05.600182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:05.600130    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:06.311094 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:06.311055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:12.443395 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:12.443365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:13.661446 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:13.661416    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"]
Sep 17 00:56:13.666361 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:13.666344    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"
Sep 17 00:56:13.668694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:13.668669    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:56:13.671346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:13.671326    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"]
Sep 17 00:56:13.700918 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:13.700893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d19ee-0a45-4d3a-89f1-5785332cb0c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr\" (UID: \"769d19ee-0a45-4d3a-89f1-5785332cb0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"
Sep 17 00:56:13.801575 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:13.801538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d19ee-0a45-4d3a-89f1-5785332cb0c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr\" (UID: \"769d19ee-0a45-4d3a-89f1-5785332cb0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"
Sep 17 00:56:13.802122 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:13.802105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d19ee-0a45-4d3a-89f1-5785332cb0c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr\" (UID: \"769d19ee-0a45-4d3a-89f1-5785332cb0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"
Sep 17 00:56:13.985192 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:13.985107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"
Sep 17 00:56:14.055828 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:14.055576    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:14.261342 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:14.261321    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"]
Sep 17 00:56:14.263616 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:56:14.263591    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769d19ee_0a45_4d3a_89f1_5785332cb0c1.slice/crio-d7c387a2e638f3a816096290d971f96954b7142ad7b101ea5de60c3b93fb6c5c WatchSource:0}: Error finding container d7c387a2e638f3a816096290d971f96954b7142ad7b101ea5de60c3b93fb6c5c: Status 404 returned error can't find the container with id d7c387a2e638f3a816096290d971f96954b7142ad7b101ea5de60c3b93fb6c5c
Sep 17 00:56:14.765156 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:14.765110    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:15.244640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:15.244607    2576 generic.go:356] "Generic (PLEG): container finished" podID="769d19ee-0a45-4d3a-89f1-5785332cb0c1" containerID="e1f6488820a265fbb8661b4dec03ed7a6d847cb26921dc5543bf18dd8b98c134" exitCode=0
Sep 17 00:56:15.244797 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:15.244702    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr" event={"ID":"769d19ee-0a45-4d3a-89f1-5785332cb0c1","Type":"ContainerDied","Data":"e1f6488820a265fbb8661b4dec03ed7a6d847cb26921dc5543bf18dd8b98c134"}
Sep 17 00:56:15.244797 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:15.244767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr" event={"ID":"769d19ee-0a45-4d3a-89f1-5785332cb0c1","Type":"ContainerStarted","Data":"d7c387a2e638f3a816096290d971f96954b7142ad7b101ea5de60c3b93fb6c5c"}
Sep 17 00:56:15.410129 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:15.410095    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:15.679394 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:15.679359    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:16.329597 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:16.329568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:16.390444 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:16.390423    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"
Sep 17 00:56:16.421362 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:16.421335    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d19ee-0a45-4d3a-89f1-5785332cb0c1-maas-api-service-ca\") pod \"769d19ee-0a45-4d3a-89f1-5785332cb0c1\" (UID: \"769d19ee-0a45-4d3a-89f1-5785332cb0c1\") "
Sep 17 00:56:16.421668 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:16.421646    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769d19ee-0a45-4d3a-89f1-5785332cb0c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "769d19ee-0a45-4d3a-89f1-5785332cb0c1" (UID: "769d19ee-0a45-4d3a-89f1-5785332cb0c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:56:16.522132 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:16.522061    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d19ee-0a45-4d3a-89f1-5785332cb0c1-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:56:17.049350 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:17.049314    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:17.257838 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:17.257804    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"
Sep 17 00:56:17.257838 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:17.257835    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr" event={"ID":"769d19ee-0a45-4d3a-89f1-5785332cb0c1","Type":"ContainerDied","Data":"d7c387a2e638f3a816096290d971f96954b7142ad7b101ea5de60c3b93fb6c5c"}
Sep 17 00:56:17.258036 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:17.257875    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7c387a2e638f3a816096290d971f96954b7142ad7b101ea5de60c3b93fb6c5c"
Sep 17 00:56:17.934577 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:17.934542    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:18.661312 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:18.661275    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:19.272395 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:19.272366    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:21.294051 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:21.294024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:29.113823 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.113790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:29.130532 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.130495    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"]
Sep 17 00:56:29.130914 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.130836    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq" podUID="eb8f9dbf-03a8-449a-87d9-aa0082042f52" containerName="payload-processing" containerID="cri-o://e231761d3a067de2697a99f40a200e7c72e6ca4f3abb4fb8d378f1c3d19b5192" gracePeriod=30
Sep 17 00:56:29.147638 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.147588    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"]
Sep 17 00:56:29.148030 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.147864    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n" podUID="8e6cbb2e-006d-4515-9707-c11828945c08" containerName="payload-pre-processing" containerID="cri-o://264b980f59b34904e7b2741c54c7931ca96d78acc29666d64cc13b7ccc30f302" gracePeriod=30
Sep 17 00:56:29.213792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.213756    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:29.287502 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.287465    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:29.299656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.299617    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"]
Sep 17 00:56:29.301752 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.301727    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7c845e1-2vvnr"]
Sep 17 00:56:29.329883 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.329854    2576 generic.go:356] "Generic (PLEG): container finished" podID="eb8f9dbf-03a8-449a-87d9-aa0082042f52" containerID="e231761d3a067de2697a99f40a200e7c72e6ca4f3abb4fb8d378f1c3d19b5192" exitCode=0
Sep 17 00:56:29.330119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.330096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq" event={"ID":"eb8f9dbf-03a8-449a-87d9-aa0082042f52","Type":"ContainerDied","Data":"e231761d3a067de2697a99f40a200e7c72e6ca4f3abb4fb8d378f1c3d19b5192"}
Sep 17 00:56:29.332006 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.331980    2576 generic.go:356] "Generic (PLEG): container finished" podID="8e6cbb2e-006d-4515-9707-c11828945c08" containerID="264b980f59b34904e7b2741c54c7931ca96d78acc29666d64cc13b7ccc30f302" exitCode=0
Sep 17 00:56:29.332084 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.332049    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n" event={"ID":"8e6cbb2e-006d-4515-9707-c11828945c08","Type":"ContainerDied","Data":"264b980f59b34904e7b2741c54c7931ca96d78acc29666d64cc13b7ccc30f302"}
Sep 17 00:56:29.343656 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.343637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:29.435338 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.435317    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:56:29.442049 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.442024    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:56:29.537041 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.537013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6cbb2e-006d-4515-9707-c11828945c08-plugins-config-volume\") pod \"8e6cbb2e-006d-4515-9707-c11828945c08\" (UID: \"8e6cbb2e-006d-4515-9707-c11828945c08\") "
Sep 17 00:56:29.537219 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.537101    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f9dbf-03a8-449a-87d9-aa0082042f52-plugins-config-volume\") pod \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\" (UID: \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\") "
Sep 17 00:56:29.537219 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.537181    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r54m\" (UniqueName: \"kubernetes.io/projected/eb8f9dbf-03a8-449a-87d9-aa0082042f52-kube-api-access-5r54m\") pod \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\" (UID: \"eb8f9dbf-03a8-449a-87d9-aa0082042f52\") "
Sep 17 00:56:29.537332 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.537224    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/8e6cbb2e-006d-4515-9707-c11828945c08-kube-api-access-h8ffd\") pod \"8e6cbb2e-006d-4515-9707-c11828945c08\" (UID: \"8e6cbb2e-006d-4515-9707-c11828945c08\") "
Sep 17 00:56:29.537624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.537490    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6cbb2e-006d-4515-9707-c11828945c08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e6cbb2e-006d-4515-9707-c11828945c08" (UID: "8e6cbb2e-006d-4515-9707-c11828945c08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:56:29.537624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.537525    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8f9dbf-03a8-449a-87d9-aa0082042f52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb8f9dbf-03a8-449a-87d9-aa0082042f52" (UID: "eb8f9dbf-03a8-449a-87d9-aa0082042f52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:56:29.537624 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.537603    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6cbb2e-006d-4515-9707-c11828945c08-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:56:29.539861 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.539839    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8f9dbf-03a8-449a-87d9-aa0082042f52-kube-api-access-5r54m" (OuterVolumeSpecName: "kube-api-access-5r54m") pod "eb8f9dbf-03a8-449a-87d9-aa0082042f52" (UID: "eb8f9dbf-03a8-449a-87d9-aa0082042f52"). InnerVolumeSpecName "kube-api-access-5r54m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:56:29.540011 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.539993    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6cbb2e-006d-4515-9707-c11828945c08-kube-api-access-h8ffd" (OuterVolumeSpecName: "kube-api-access-h8ffd") pod "8e6cbb2e-006d-4515-9707-c11828945c08" (UID: "8e6cbb2e-006d-4515-9707-c11828945c08"). InnerVolumeSpecName "kube-api-access-h8ffd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:56:29.638781 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.638750    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r54m\" (UniqueName: \"kubernetes.io/projected/eb8f9dbf-03a8-449a-87d9-aa0082042f52-kube-api-access-5r54m\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:56:29.638781 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.638779    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/8e6cbb2e-006d-4515-9707-c11828945c08-kube-api-access-h8ffd\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:56:29.638906 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.638795    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f9dbf-03a8-449a-87d9-aa0082042f52-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:56:29.721359 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:29.721304    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:30.340119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.340087    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"
Sep 17 00:56:30.340552 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.340090    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n" event={"ID":"8e6cbb2e-006d-4515-9707-c11828945c08","Type":"ContainerDied","Data":"f1fc46af445607f26c3aa0fd1631749701fd56bd4c35c7a778b88a2ca27bf629"}
Sep 17 00:56:30.340552 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.340311    2576 scope.go:117] "RemoveContainer" containerID="264b980f59b34904e7b2741c54c7931ca96d78acc29666d64cc13b7ccc30f302"
Sep 17 00:56:30.342187 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.341793    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"
Sep 17 00:56:30.342187 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.341815    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq" event={"ID":"eb8f9dbf-03a8-449a-87d9-aa0082042f52","Type":"ContainerDied","Data":"dfa89419e6fa0bacbff9d0ee7231f1aa42fedeecfe703d8d77697fd7fc5829fa"}
Sep 17 00:56:30.354216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.354191    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:30.354314 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.354293    2576 scope.go:117] "RemoveContainer" containerID="e231761d3a067de2697a99f40a200e7c72e6ca4f3abb4fb8d378f1c3d19b5192"
Sep 17 00:56:30.369932 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.369911    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"]
Sep 17 00:56:30.373879 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.373859    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7c845e1-549d589686-9kj9n"]
Sep 17 00:56:30.383278 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.383259    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"]
Sep 17 00:56:30.388734 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.388714    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7c845e1-84c794797c-r2dtq"]
Sep 17 00:56:30.587891 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.587864    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="769d19ee-0a45-4d3a-89f1-5785332cb0c1" path="/var/lib/kubelet/pods/769d19ee-0a45-4d3a-89f1-5785332cb0c1/volumes"
Sep 17 00:56:30.588234 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.588220    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6cbb2e-006d-4515-9707-c11828945c08" path="/var/lib/kubelet/pods/8e6cbb2e-006d-4515-9707-c11828945c08/volumes"
Sep 17 00:56:30.588533 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.588522    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8f9dbf-03a8-449a-87d9-aa0082042f52" path="/var/lib/kubelet/pods/eb8f9dbf-03a8-449a-87d9-aa0082042f52/volumes"
Sep 17 00:56:30.993794 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:30.993722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:31.764750 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:31.764722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:32.368935 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:32.368906    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:32.469255 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:32.469213    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:33.061650 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:33.061613    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:37.455815 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:37.455780    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:43.589404 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589370    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"]
Sep 17 00:56:43.589815 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589795    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6cbb2e-006d-4515-9707-c11828945c08" containerName="payload-pre-processing"
Sep 17 00:56:43.589815 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589814    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6cbb2e-006d-4515-9707-c11828945c08" containerName="payload-pre-processing"
Sep 17 00:56:43.590003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589827    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="769d19ee-0a45-4d3a-89f1-5785332cb0c1" containerName="revoke-keys"
Sep 17 00:56:43.590003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589833    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="769d19ee-0a45-4d3a-89f1-5785332cb0c1" containerName="revoke-keys"
Sep 17 00:56:43.590003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589843    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8f9dbf-03a8-449a-87d9-aa0082042f52" containerName="payload-processing"
Sep 17 00:56:43.590003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589849    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8f9dbf-03a8-449a-87d9-aa0082042f52" containerName="payload-processing"
Sep 17 00:56:43.590003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589928    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="769d19ee-0a45-4d3a-89f1-5785332cb0c1" containerName="revoke-keys"
Sep 17 00:56:43.590003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589939    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6cbb2e-006d-4515-9707-c11828945c08" containerName="payload-pre-processing"
Sep 17 00:56:43.590003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.589945    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8f9dbf-03a8-449a-87d9-aa0082042f52" containerName="payload-processing"
Sep 17 00:56:43.593153 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.593120    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.596762 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.596740    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"kube-root-ca.crt\""
Sep 17 00:56:43.596870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.596805    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"auth-test-model-bcd112-kserve-self-signed-certs\""
Sep 17 00:56:43.596870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.596768    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"openshift-service-ca.crt\""
Sep 17 00:56:43.596870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.596762    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"default-dockercfg-qxzgf\""
Sep 17 00:56:43.603877 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.603854    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"]
Sep 17 00:56:43.655920 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.655892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw28c\" (UniqueName: \"kubernetes.io/projected/e3d04bf9-836a-4732-a9b0-ca04648e61a9-kube-api-access-sw28c\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.656057 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.655929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tls-certs\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.656057 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.655958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tmp-dir\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.656057 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.656048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-dshm\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.656235 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.656116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-home\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.656235 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.656132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-model-cache\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.714290 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.714261    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:43.757513 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.757486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-home\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.757513 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.757511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-model-cache\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.757675 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.757557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw28c\" (UniqueName: \"kubernetes.io/projected/e3d04bf9-836a-4732-a9b0-ca04648e61a9-kube-api-access-sw28c\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.757675 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.757639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tls-certs\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.757779 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.757690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tmp-dir\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.757830 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.757799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-dshm\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.757953 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.757934    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-model-cache\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.758005 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.757951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-home\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.758048 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.758008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tmp-dir\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.759980 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.759956    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-dshm\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.760244 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.760228    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tls-certs\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.765912 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.765891    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw28c\" (UniqueName: \"kubernetes.io/projected/e3d04bf9-836a-4732-a9b0-ca04648e61a9-kube-api-access-sw28c\") pod \"auth-test-model-bcd112-kserve-664477f86c-qqtlp\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:43.903911 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:43.903829    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:44.175671 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:44.175650    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"]
Sep 17 00:56:44.177472 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:56:44.177443    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3d04bf9_836a_4732_a9b0_ca04648e61a9.slice/crio-8787792da12c51971158d0a8bb3504665a62fc9d870332624907a4a59bef3d94 WatchSource:0}: Error finding container 8787792da12c51971158d0a8bb3504665a62fc9d870332624907a4a59bef3d94: Status 404 returned error can't find the container with id 8787792da12c51971158d0a8bb3504665a62fc9d870332624907a4a59bef3d94
Sep 17 00:56:44.422846 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:44.422816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp" event={"ID":"e3d04bf9-836a-4732-a9b0-ca04648e61a9","Type":"ContainerStarted","Data":"9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413"}
Sep 17 00:56:44.423035 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:44.422856    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp" event={"ID":"e3d04bf9-836a-4732-a9b0-ca04648e61a9","Type":"ContainerStarted","Data":"8787792da12c51971158d0a8bb3504665a62fc9d870332624907a4a59bef3d94"}
Sep 17 00:56:44.441626 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:44.441532    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp" podStartSLOduration=1.441513595 podStartE2EDuration="1.441513595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:56:44.438104163 +0000 UTC m=+1600.393700504" watchObservedRunningTime="2026-09-17 00:56:44.441513595 +0000 UTC m=+1600.397109936"
Sep 17 00:56:53.904633 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:53.904597    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:53.905064 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:53.904650    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:53.917310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:53.917289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:54.492003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:54.491977    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 00:56:58.895296 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.895214    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"]
Sep 17 00:56:58.899261 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.899241    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:58.901586 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.901567    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"embedding-w0-a79f56-kserve-self-signed-certs\""
Sep 17 00:56:58.906089 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.906067    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"]
Sep 17 00:56:58.983737 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.983706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-dshm\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:58.983737 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.983740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-home\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:58.983949 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.983902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-model-cache\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:58.983989 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.983943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpn9z\" (UniqueName: \"kubernetes.io/projected/966a4e59-4cc2-4eb9-a0ea-70f855272550-kube-api-access-jpn9z\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:58.984138 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.984111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966a4e59-4cc2-4eb9-a0ea-70f855272550-tls-certs\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:58.984223 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.984207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-tmp-dir\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:58.995905 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:58.995864    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:56:59.084697 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.084670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966a4e59-4cc2-4eb9-a0ea-70f855272550-tls-certs\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.084792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.084705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-tmp-dir\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.084792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.084737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-dshm\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.084792 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.084755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-home\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.084957 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.084831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-model-cache\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.084957 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.084852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpn9z\" (UniqueName: \"kubernetes.io/projected/966a4e59-4cc2-4eb9-a0ea-70f855272550-kube-api-access-jpn9z\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.085281 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.085252    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-tmp-dir\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.085281 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.085275    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-model-cache\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.086120 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.086090    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-home\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.087880 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.087854    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-dshm\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.088017 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.087998    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966a4e59-4cc2-4eb9-a0ea-70f855272550-tls-certs\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.095659 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.095638    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpn9z\" (UniqueName: \"kubernetes.io/projected/966a4e59-4cc2-4eb9-a0ea-70f855272550-kube-api-access-jpn9z\") pod \"embedding-w0-a79f56-kserve-7678689584-7n8jp\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") " pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.209514 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.209433    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:56:59.484902 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.484879    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"]
Sep 17 00:56:59.486603 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:56:59.486563    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod966a4e59_4cc2_4eb9_a0ea_70f855272550.slice/crio-534a3e2207d483538cf4146b1902fc045c403b63cb7e09417b93313fe99d9354 WatchSource:0}: Error finding container 534a3e2207d483538cf4146b1902fc045c403b63cb7e09417b93313fe99d9354: Status 404 returned error can't find the container with id 534a3e2207d483538cf4146b1902fc045c403b63cb7e09417b93313fe99d9354
Sep 17 00:56:59.510928 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:56:59.510900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp" event={"ID":"966a4e59-4cc2-4eb9-a0ea-70f855272550","Type":"ContainerStarted","Data":"534a3e2207d483538cf4146b1902fc045c403b63cb7e09417b93313fe99d9354"}
Sep 17 00:57:00.518545 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:00.518513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp" event={"ID":"966a4e59-4cc2-4eb9-a0ea-70f855272550","Type":"ContainerStarted","Data":"c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21"}
Sep 17 00:57:00.534734 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:00.534681    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp" podStartSLOduration=2.534664335 podStartE2EDuration="2.534664335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:57:00.532261415 +0000 UTC m=+1616.487857770" watchObservedRunningTime="2026-09-17 00:57:00.534664335 +0000 UTC m=+1616.490260675"
Sep 17 00:57:04.778533 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:04.778504    2576 scope.go:117] "RemoveContainer" containerID="8d51db2a57367f6c9737580dc6a1c659ea06dd7923df9479fc8ae34ed2196cb1"
Sep 17 00:57:04.786997 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:04.786982    2576 scope.go:117] "RemoveContainer" containerID="b285be96f8ba445ebeb959e78638d22678b924dacc51fbce92f97e89269b9767"
Sep 17 00:57:09.210489 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:09.210453    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:57:09.210967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:09.210612    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:57:09.223289 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:09.223265    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:57:09.586688 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:09.586659    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:57:13.828019 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:13.827990    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:14.431722 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:14.431690    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:19.926803 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:19.926769    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:25.804768 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:25.804728    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"]
Sep 17 00:57:25.805193 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:25.804960    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy" containerID="cri-o://b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483" gracePeriod=30
Sep 17 00:57:25.944201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:25.944135    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:26.044447 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:26.044410    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:26.999107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:26.999076    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:27.097568 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:27.097535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:27.197620 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:27.197591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:27.796241 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:27.796208    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:28.178827 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:28.178741    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:28.281326 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:28.281293    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:29.257253 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:29.257215    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:31.064478 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.064418    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:57:31.188436 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.188405    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-envoy\") pod \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") "
Sep 17 00:57:31.188625 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.188451    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istiod-ca-cert\") pod \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") "
Sep 17 00:57:31.188625 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.188488    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-podinfo\") pod \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") "
Sep 17 00:57:31.188625 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.188505    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-token\") pod \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") "
Sep 17 00:57:31.188625 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.188575    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-data\") pod \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") "
Sep 17 00:57:31.188625 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.188623    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-socket\") pod \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") "
Sep 17 00:57:31.188904 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.188673    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4x5b\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-kube-api-access-b4x5b\") pod \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") "
Sep 17 00:57:31.188904 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.188700    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-credential-socket\") pod \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") "
Sep 17 00:57:31.188904 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.188725    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-certs\") pod \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\" (UID: \"21ef85c3-30c3-4dde-b8d2-620370b4f53e\") "
Sep 17 00:57:31.189101 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.189033    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21ef85c3-30c3-4dde-b8d2-620370b4f53e" (UID: "21ef85c3-30c3-4dde-b8d2-620370b4f53e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:57:31.189212 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.189048    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21ef85c3-30c3-4dde-b8d2-620370b4f53e" (UID: "21ef85c3-30c3-4dde-b8d2-620370b4f53e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:57:31.189340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.189311    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21ef85c3-30c3-4dde-b8d2-620370b4f53e" (UID: "21ef85c3-30c3-4dde-b8d2-620370b4f53e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:57:31.189340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.189321    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21ef85c3-30c3-4dde-b8d2-620370b4f53e" (UID: "21ef85c3-30c3-4dde-b8d2-620370b4f53e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:57:31.189525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.189482    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-data" (OuterVolumeSpecName: "istio-data") pod "21ef85c3-30c3-4dde-b8d2-620370b4f53e" (UID: "21ef85c3-30c3-4dde-b8d2-620370b4f53e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:57:31.190848 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.190818    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21ef85c3-30c3-4dde-b8d2-620370b4f53e" (UID: "21ef85c3-30c3-4dde-b8d2-620370b4f53e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:57:31.190947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.190896    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21ef85c3-30c3-4dde-b8d2-620370b4f53e" (UID: "21ef85c3-30c3-4dde-b8d2-620370b4f53e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:57:31.190991 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.190949    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-token" (OuterVolumeSpecName: "istio-token") pod "21ef85c3-30c3-4dde-b8d2-620370b4f53e" (UID: "21ef85c3-30c3-4dde-b8d2-620370b4f53e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:57:31.191346 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.191320    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-kube-api-access-b4x5b" (OuterVolumeSpecName: "kube-api-access-b4x5b") pod "21ef85c3-30c3-4dde-b8d2-620370b4f53e" (UID: "21ef85c3-30c3-4dde-b8d2-620370b4f53e"). InnerVolumeSpecName "kube-api-access-b4x5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:57:31.295201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.290940    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:57:31.295201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.290975    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:57:31.295201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.290988    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:57:31.295201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.291002    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:57:31.295201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.291020    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4x5b\" (UniqueName: \"kubernetes.io/projected/21ef85c3-30c3-4dde-b8d2-620370b4f53e-kube-api-access-b4x5b\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:57:31.295201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.291036    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:57:31.295201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.291051    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:57:31.295201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.291064    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:57:31.295201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.291081    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ef85c3-30c3-4dde-b8d2-620370b4f53e-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:57:31.702974 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.702935    2576 generic.go:356] "Generic (PLEG): container finished" podID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerID="b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483" exitCode=0
Sep 17 00:57:31.703245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.703014    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"
Sep 17 00:57:31.703245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.703017    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" event={"ID":"21ef85c3-30c3-4dde-b8d2-620370b4f53e","Type":"ContainerDied","Data":"b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483"}
Sep 17 00:57:31.703245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.703074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m" event={"ID":"21ef85c3-30c3-4dde-b8d2-620370b4f53e","Type":"ContainerDied","Data":"13ad0c93ee1f851c4c4216643e959340af5ac393a8486655c56c099cb1e77796"}
Sep 17 00:57:31.703245 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.703097    2576 scope.go:117] "RemoveContainer" containerID="b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483"
Sep 17 00:57:31.713240 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.713213    2576 scope.go:117] "RemoveContainer" containerID="f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6"
Sep 17 00:57:31.722078 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.722060    2576 scope.go:117] "RemoveContainer" containerID="b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483"
Sep 17 00:57:31.722634 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:57:31.722607    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483\": container with ID starting with b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483 not found: ID does not exist" containerID="b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483"
Sep 17 00:57:31.722747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.722645    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483"} err="failed to get container status \"b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483\": rpc error: code = NotFound desc = could not find container \"b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483\": container with ID starting with b06f7264912fe6320010122b6949accd70d7f833566ad4c4dbd6c5e133f8f483 not found: ID does not exist"
Sep 17 00:57:31.722747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.722670    2576 scope.go:117] "RemoveContainer" containerID="f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6"
Sep 17 00:57:31.723060 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:57:31.723018    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6\": container with ID starting with f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6 not found: ID does not exist" containerID="f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6"
Sep 17 00:57:31.723191 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.723057    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6"} err="failed to get container status \"f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6\": rpc error: code = NotFound desc = could not find container \"f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6\": container with ID starting with f633e62fbc3c3cf0c1dd7cb10583241a4b80b244a9363b04ac6c41c04bd222e6 not found: ID does not exist"
Sep 17 00:57:31.725409 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.725386    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"]
Sep 17 00:57:31.728070 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:31.728050    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c7c845e1-openshift-default-78b49595fb-cd97m"]
Sep 17 00:57:32.588606 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:32.588575    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" path="/var/lib/kubelet/pods/21ef85c3-30c3-4dde-b8d2-620370b4f53e/volumes"
Sep 17 00:57:34.988137 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:34.988106    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:35.040571 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:35.040537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:35.659408 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:35.659355    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:41.495012 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:41.494960    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:41.602506 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:41.602470    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:41.695560 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:41.695528    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:52.187710 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:52.187666    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:52.401554 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:52.401523    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:58.520185 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:58.520135    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:57:58.567495 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:57:58.567463    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:01.066455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.066417    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:01.167833 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.167796    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"]
Sep 17 00:58:01.168264 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.168251    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy"
Sep 17 00:58:01.168316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.168266    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy"
Sep 17 00:58:01.168316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.168278    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy"
Sep 17 00:58:01.168316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.168283    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy"
Sep 17 00:58:01.168413 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.168352    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy"
Sep 17 00:58:01.171490 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.171475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"
Sep 17 00:58:01.173793 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.173772    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:58:01.180067 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.179563    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"]
Sep 17 00:58:01.295614 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.295579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4b38-9876-4234-9da0-674c859023c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6\" (UID: \"7bbd4b38-9876-4234-9da0-674c859023c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"
Sep 17 00:58:01.397232 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.397112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4b38-9876-4234-9da0-674c859023c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6\" (UID: \"7bbd4b38-9876-4234-9da0-674c859023c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"
Sep 17 00:58:01.397740 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.397722    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4b38-9876-4234-9da0-674c859023c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6\" (UID: \"7bbd4b38-9876-4234-9da0-674c859023c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"
Sep 17 00:58:01.458911 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.458874    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:01.483826 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.483793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"
Sep 17 00:58:01.760067 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.760044    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"]
Sep 17 00:58:01.761824 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:58:01.761798    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbd4b38_9876_4234_9da0_674c859023c4.slice/crio-0e8af8065a57962a1eeb9a7ee395c30c8c7bc114d74f5723d2701b68ae759630 WatchSource:0}: Error finding container 0e8af8065a57962a1eeb9a7ee395c30c8c7bc114d74f5723d2701b68ae759630: Status 404 returned error can't find the container with id 0e8af8065a57962a1eeb9a7ee395c30c8c7bc114d74f5723d2701b68ae759630
Sep 17 00:58:01.886834 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:01.886800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6" event={"ID":"7bbd4b38-9876-4234-9da0-674c859023c4","Type":"ContainerStarted","Data":"0e8af8065a57962a1eeb9a7ee395c30c8c7bc114d74f5723d2701b68ae759630"}
Sep 17 00:58:02.261158 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:02.261122    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:02.883842 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:02.883812    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:02.894024 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:02.893990    2576 generic.go:356] "Generic (PLEG): container finished" podID="7bbd4b38-9876-4234-9da0-674c859023c4" containerID="22cd6cc0eeba7c33bf11ca58c238876aa0a2b81a2081e87295a868b3e123c156" exitCode=0
Sep 17 00:58:02.894208 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:02.894050    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6" event={"ID":"7bbd4b38-9876-4234-9da0-674c859023c4","Type":"ContainerDied","Data":"22cd6cc0eeba7c33bf11ca58c238876aa0a2b81a2081e87295a868b3e123c156"}
Sep 17 00:58:03.660880 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:03.660844    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:04.044490 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.044463    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"
Sep 17 00:58:04.128114 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.128076    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4b38-9876-4234-9da0-674c859023c4-maas-api-service-ca\") pod \"7bbd4b38-9876-4234-9da0-674c859023c4\" (UID: \"7bbd4b38-9876-4234-9da0-674c859023c4\") "
Sep 17 00:58:04.128488 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.128462    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbd4b38-9876-4234-9da0-674c859023c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bbd4b38-9876-4234-9da0-674c859023c4" (UID: "7bbd4b38-9876-4234-9da0-674c859023c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:58:04.229314 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.229228    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4b38-9876-4234-9da0-674c859023c4-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:04.342537 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.342505    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:04.823178 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.818854    2576 scope.go:117] "RemoveContainer" containerID="532b835f2c07a1f54f8f5d516e78f61d9928969e8e9a74bb0832123054d3bec2"
Sep 17 00:58:04.909219 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.909187    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"
Sep 17 00:58:04.909219 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.909200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6" event={"ID":"7bbd4b38-9876-4234-9da0-674c859023c4","Type":"ContainerDied","Data":"0e8af8065a57962a1eeb9a7ee395c30c8c7bc114d74f5723d2701b68ae759630"}
Sep 17 00:58:04.909414 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.909277    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e8af8065a57962a1eeb9a7ee395c30c8c7bc114d74f5723d2701b68ae759630"
Sep 17 00:58:04.976670 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:04.976636    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:05.734166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:05.734112    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:06.148982 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:06.148945    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"]
Sep 17 00:58:06.158076 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:06.158043    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"]
Sep 17 00:58:06.162241 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:06.162201    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"]
Sep 17 00:58:06.240770 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:06.240739    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:13.272758 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:13.272725    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:16.753684 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.753651    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:16.775274 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.775236    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"]
Sep 17 00:58:16.784825 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.775921    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx" podUID="e09dedfe-73ef-46b8-bf08-d037278c201c" containerName="payload-processing" containerID="cri-o://2ca321f051f8de4f6b08040eac3ce211f344c7b2baefb3001724d4c164203de8" gracePeriod=30
Sep 17 00:58:16.786718 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.786686    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"]
Sep 17 00:58:16.787264 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.787209    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm" podUID="13d9df51-4813-44ca-866a-8f2b0c3b4941" containerName="payload-pre-processing" containerID="cri-o://fa4bafc2e3b6b108fcb472575a250a99f0cc08fcc743cdc3f384aca2f671db96" gracePeriod=30
Sep 17 00:58:16.817377 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.817284    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:16.873121 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.873091    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:16.936023 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.935985    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"]
Sep 17 00:58:16.939818 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.939718    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:16.943991 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.943949    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a79f56-rp6d6"]
Sep 17 00:58:16.985458 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.985428    2576 generic.go:356] "Generic (PLEG): container finished" podID="13d9df51-4813-44ca-866a-8f2b0c3b4941" containerID="fa4bafc2e3b6b108fcb472575a250a99f0cc08fcc743cdc3f384aca2f671db96" exitCode=0
Sep 17 00:58:16.985605 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.985497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm" event={"ID":"13d9df51-4813-44ca-866a-8f2b0c3b4941","Type":"ContainerDied","Data":"fa4bafc2e3b6b108fcb472575a250a99f0cc08fcc743cdc3f384aca2f671db96"}
Sep 17 00:58:16.987475 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.987451    2576 generic.go:356] "Generic (PLEG): container finished" podID="e09dedfe-73ef-46b8-bf08-d037278c201c" containerID="2ca321f051f8de4f6b08040eac3ce211f344c7b2baefb3001724d4c164203de8" exitCode=0
Sep 17 00:58:16.987591 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:16.987521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx" event={"ID":"e09dedfe-73ef-46b8-bf08-d037278c201c","Type":"ContainerDied","Data":"2ca321f051f8de4f6b08040eac3ce211f344c7b2baefb3001724d4c164203de8"}
Sep 17 00:58:17.069895 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.069872    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:58:17.097332 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.097305    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:58:17.158944 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.158912    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdjr6\" (UniqueName: \"kubernetes.io/projected/13d9df51-4813-44ca-866a-8f2b0c3b4941-kube-api-access-fdjr6\") pod \"13d9df51-4813-44ca-866a-8f2b0c3b4941\" (UID: \"13d9df51-4813-44ca-866a-8f2b0c3b4941\") "
Sep 17 00:58:17.159116 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.159082    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d9df51-4813-44ca-866a-8f2b0c3b4941-plugins-config-volume\") pod \"13d9df51-4813-44ca-866a-8f2b0c3b4941\" (UID: \"13d9df51-4813-44ca-866a-8f2b0c3b4941\") "
Sep 17 00:58:17.159482 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.159449    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d9df51-4813-44ca-866a-8f2b0c3b4941-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13d9df51-4813-44ca-866a-8f2b0c3b4941" (UID: "13d9df51-4813-44ca-866a-8f2b0c3b4941"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:58:17.161199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.161180    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d9df51-4813-44ca-866a-8f2b0c3b4941-kube-api-access-fdjr6" (OuterVolumeSpecName: "kube-api-access-fdjr6") pod "13d9df51-4813-44ca-866a-8f2b0c3b4941" (UID: "13d9df51-4813-44ca-866a-8f2b0c3b4941"). InnerVolumeSpecName "kube-api-access-fdjr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:58:17.260096 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.260008    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09dedfe-73ef-46b8-bf08-d037278c201c-plugins-config-volume\") pod \"e09dedfe-73ef-46b8-bf08-d037278c201c\" (UID: \"e09dedfe-73ef-46b8-bf08-d037278c201c\") "
Sep 17 00:58:17.260096 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.260074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkr94\" (UniqueName: \"kubernetes.io/projected/e09dedfe-73ef-46b8-bf08-d037278c201c-kube-api-access-qkr94\") pod \"e09dedfe-73ef-46b8-bf08-d037278c201c\" (UID: \"e09dedfe-73ef-46b8-bf08-d037278c201c\") "
Sep 17 00:58:17.260363 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.260345    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d9df51-4813-44ca-866a-8f2b0c3b4941-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:17.260407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.260371    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdjr6\" (UniqueName: \"kubernetes.io/projected/13d9df51-4813-44ca-866a-8f2b0c3b4941-kube-api-access-fdjr6\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:17.260450 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.260407    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09dedfe-73ef-46b8-bf08-d037278c201c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e09dedfe-73ef-46b8-bf08-d037278c201c" (UID: "e09dedfe-73ef-46b8-bf08-d037278c201c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:58:17.262331 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.262311    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09dedfe-73ef-46b8-bf08-d037278c201c-kube-api-access-qkr94" (OuterVolumeSpecName: "kube-api-access-qkr94") pod "e09dedfe-73ef-46b8-bf08-d037278c201c" (UID: "e09dedfe-73ef-46b8-bf08-d037278c201c"). InnerVolumeSpecName "kube-api-access-qkr94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:58:17.278132 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.278102    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:17.360893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.360852    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09dedfe-73ef-46b8-bf08-d037278c201c-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:17.360893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.360886    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkr94\" (UniqueName: \"kubernetes.io/projected/e09dedfe-73ef-46b8-bf08-d037278c201c-kube-api-access-qkr94\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:17.375024 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.374995    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:17.994646 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.994616    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"
Sep 17 00:58:17.995103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.994647    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx" event={"ID":"e09dedfe-73ef-46b8-bf08-d037278c201c","Type":"ContainerDied","Data":"38edbe1caa6bb88c8968070df5f9bcd26529a3835f887559fd3f9ab08b4041ff"}
Sep 17 00:58:17.995103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.994744    2576 scope.go:117] "RemoveContainer" containerID="2ca321f051f8de4f6b08040eac3ce211f344c7b2baefb3001724d4c164203de8"
Sep 17 00:58:17.996240 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.996218    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm" event={"ID":"13d9df51-4813-44ca-866a-8f2b0c3b4941","Type":"ContainerDied","Data":"b33a83869b985b961da4c472393260a08b3463cab428dfb1700b1317f9a8d9a4"}
Sep 17 00:58:17.996325 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:17.996251    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"
Sep 17 00:58:18.005662 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.005641    2576 scope.go:117] "RemoveContainer" containerID="fa4bafc2e3b6b108fcb472575a250a99f0cc08fcc743cdc3f384aca2f671db96"
Sep 17 00:58:18.019984 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.019955    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"]
Sep 17 00:58:18.023881 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.023863    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a79f56-574c6f9567-n6dwx"]
Sep 17 00:58:18.032722 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.032703    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"]
Sep 17 00:58:18.040879 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.040857    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a79f56-574d7f5798-z7ltm"]
Sep 17 00:58:18.086460 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.086406    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:18.588395 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.588356    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d9df51-4813-44ca-866a-8f2b0c3b4941" path="/var/lib/kubelet/pods/13d9df51-4813-44ca-866a-8f2b0c3b4941/volumes"
Sep 17 00:58:18.588788 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.588768    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bbd4b38-9876-4234-9da0-674c859023c4" path="/var/lib/kubelet/pods/7bbd4b38-9876-4234-9da0-674c859023c4/volumes"
Sep 17 00:58:18.589195 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.589177    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e09dedfe-73ef-46b8-bf08-d037278c201c" path="/var/lib/kubelet/pods/e09dedfe-73ef-46b8-bf08-d037278c201c/volumes"
Sep 17 00:58:18.848537 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:18.848462    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:19.160134 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.160051    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:19.261492 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.261467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:19.489378 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.489307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:19.588919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.588893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:19.751061 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.750946    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"]
Sep 17 00:58:19.751533 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751510    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bbd4b38-9876-4234-9da0-674c859023c4" containerName="revoke-keys"
Sep 17 00:58:19.751533 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751534    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bbd4b38-9876-4234-9da0-674c859023c4" containerName="revoke-keys"
Sep 17 00:58:19.751721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751562    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09dedfe-73ef-46b8-bf08-d037278c201c" containerName="payload-processing"
Sep 17 00:58:19.751721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751568    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09dedfe-73ef-46b8-bf08-d037278c201c" containerName="payload-processing"
Sep 17 00:58:19.751721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751578    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d9df51-4813-44ca-866a-8f2b0c3b4941" containerName="payload-pre-processing"
Sep 17 00:58:19.751721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751583    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d9df51-4813-44ca-866a-8f2b0c3b4941" containerName="payload-pre-processing"
Sep 17 00:58:19.751721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751644    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bbd4b38-9876-4234-9da0-674c859023c4" containerName="revoke-keys"
Sep 17 00:58:19.751721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751653    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e09dedfe-73ef-46b8-bf08-d037278c201c" containerName="payload-processing"
Sep 17 00:58:19.751721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751661    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ef85c3-30c3-4dde-b8d2-620370b4f53e" containerName="istio-proxy"
Sep 17 00:58:19.751721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.751669    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d9df51-4813-44ca-866a-8f2b0c3b4941" containerName="payload-pre-processing"
Sep 17 00:58:19.756378 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.756360    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.758810 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.758791    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-80e99a-openshift-default-dockercfg-r2zpv\""
Sep 17 00:58:19.766194 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.766173    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"]
Sep 17 00:58:19.777811 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.777790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:19.876265 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.876238    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:19.883407 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.883389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-data\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.883509 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.883421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-istio-token\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.883509 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.883442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdkq\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-kube-api-access-gjdkq\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.883509 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.883493    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-certs\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.883670 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.883524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-envoy\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.883670 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.883621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-socket\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.883773 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.883668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2356fc31-e2c5-4309-9742-62d620edd243-istiod-ca-cert\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.883773 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.883741    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2356fc31-e2c5-4309-9742-62d620edd243-istio-podinfo\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.883873 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.883772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-credential-socket\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985256 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-data\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985419 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-istio-token\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985419 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdkq\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-kube-api-access-gjdkq\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985419 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-certs\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985419 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-envoy\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985419 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-socket\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2356fc31-e2c5-4309-9742-62d620edd243-istiod-ca-cert\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2356fc31-e2c5-4309-9742-62d620edd243-istio-podinfo\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-credential-socket\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-data\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.985694 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-certs\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.986014 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.985723    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-socket\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.986101 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.986080    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2356fc31-e2c5-4309-9742-62d620edd243-istiod-ca-cert\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.986198 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.986184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-credential-socket\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.988087 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.988058    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2356fc31-e2c5-4309-9742-62d620edd243-istio-podinfo\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.988240 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.988223    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-envoy\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.995689 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.995664    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-istio-token\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:19.995842 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:19.995823    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdkq\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-kube-api-access-gjdkq\") pod \"e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") " pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:20.069659 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:20.069635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:20.306916 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:20.306884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:20.349182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:20.349157    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"]
Sep 17 00:58:20.350436 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:58:20.350409    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2356fc31_e2c5_4309_9742_62d620edd243.slice/crio-cd82d0099b4a0e357def989715e88227cc1ebfcc59ff20d36dbd56095c8756d9 WatchSource:0}: Error finding container cd82d0099b4a0e357def989715e88227cc1ebfcc59ff20d36dbd56095c8756d9: Status 404 returned error can't find the container with id cd82d0099b4a0e357def989715e88227cc1ebfcc59ff20d36dbd56095c8756d9
Sep 17 00:58:20.353263 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:20.353231    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:58:20.353377 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:20.353310    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:58:20.353377 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:20.353367    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:58:21.016827 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.016793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" event={"ID":"2356fc31-e2c5-4309-9742-62d620edd243","Type":"ContainerStarted","Data":"72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8"}
Sep 17 00:58:21.016989 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.016844    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" event={"ID":"2356fc31-e2c5-4309-9742-62d620edd243","Type":"ContainerStarted","Data":"cd82d0099b4a0e357def989715e88227cc1ebfcc59ff20d36dbd56095c8756d9"}
Sep 17 00:58:21.039705 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.039652    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" podStartSLOduration=2.039635451 podStartE2EDuration="2.039635451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:58:21.039286119 +0000 UTC m=+1696.994882461" watchObservedRunningTime="2026-09-17 00:58:21.039635451 +0000 UTC m=+1696.995231793"
Sep 17 00:58:21.070554 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.070531    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:21.076019 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.076001    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:21.170481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.170450    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9" podUID="2d2469d0-da52-435b-bc40-43a27fc06195" containerName="main" containerID="cri-o://d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36" gracePeriod=45
Sep 17 00:58:21.181535 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.181506    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp" podUID="966a4e59-4cc2-4eb9-a0ea-70f855272550" containerName="main" containerID="cri-o://c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21" gracePeriod=45
Sep 17 00:58:21.182202 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.182172    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt" podUID="e4110dd9-03ae-4f71-b4bc-bb08c9951629" containerName="main" containerID="cri-o://9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416" gracePeriod=45
Sep 17 00:58:21.433918 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.433893    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:58:21.496751 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.496729    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:58:21.499930 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.499905    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:58:21.502050 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502030    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-home\") pod \"2d2469d0-da52-435b-bc40-43a27fc06195\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") "
Sep 17 00:58:21.502206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502064    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-home\") pod \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") "
Sep 17 00:58:21.502206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502097    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-model-cache\") pod \"966a4e59-4cc2-4eb9-a0ea-70f855272550\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") "
Sep 17 00:58:21.502206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502117    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-dshm\") pod \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") "
Sep 17 00:58:21.502206 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502197    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-dshm\") pod \"966a4e59-4cc2-4eb9-a0ea-70f855272550\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") "
Sep 17 00:58:21.502438 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502359    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-home" (OuterVolumeSpecName: "home") pod "2d2469d0-da52-435b-bc40-43a27fc06195" (UID: "2d2469d0-da52-435b-bc40-43a27fc06195"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.502438 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502349    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-home" (OuterVolumeSpecName: "home") pod "e4110dd9-03ae-4f71-b4bc-bb08c9951629" (UID: "e4110dd9-03ae-4f71-b4bc-bb08c9951629"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.502438 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502428    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-model-cache" (OuterVolumeSpecName: "model-cache") pod "966a4e59-4cc2-4eb9-a0ea-70f855272550" (UID: "966a4e59-4cc2-4eb9-a0ea-70f855272550"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.502652 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502634    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-model-cache\") pod \"2d2469d0-da52-435b-bc40-43a27fc06195\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") "
Sep 17 00:58:21.502721 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502671    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d2469d0-da52-435b-bc40-43a27fc06195" (UID: "2d2469d0-da52-435b-bc40-43a27fc06195"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.502816 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502801    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tmp-dir\") pod \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") "
Sep 17 00:58:21.502938 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502924    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-tmp-dir\") pod \"2d2469d0-da52-435b-bc40-43a27fc06195\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") "
Sep 17 00:58:21.503005 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502955    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpsz4\" (UniqueName: \"kubernetes.io/projected/2d2469d0-da52-435b-bc40-43a27fc06195-kube-api-access-bpsz4\") pod \"2d2469d0-da52-435b-bc40-43a27fc06195\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") "
Sep 17 00:58:21.503005 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.502986    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4110dd9-03ae-4f71-b4bc-bb08c9951629" (UID: "e4110dd9-03ae-4f71-b4bc-bb08c9951629"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.503199 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503179    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d2469d0-da52-435b-bc40-43a27fc06195" (UID: "2d2469d0-da52-435b-bc40-43a27fc06195"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.503267 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503184    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2469d0-da52-435b-bc40-43a27fc06195-tls-certs\") pod \"2d2469d0-da52-435b-bc40-43a27fc06195\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") "
Sep 17 00:58:21.503267 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503241    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tls-certs\") pod \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") "
Sep 17 00:58:21.503379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503269    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-home\") pod \"966a4e59-4cc2-4eb9-a0ea-70f855272550\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") "
Sep 17 00:58:21.503379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503297    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-model-cache\") pod \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") "
Sep 17 00:58:21.503379 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503346    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpn9z\" (UniqueName: \"kubernetes.io/projected/966a4e59-4cc2-4eb9-a0ea-70f855272550-kube-api-access-jpn9z\") pod \"966a4e59-4cc2-4eb9-a0ea-70f855272550\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") "
Sep 17 00:58:21.503525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503377    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-tmp-dir\") pod \"966a4e59-4cc2-4eb9-a0ea-70f855272550\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") "
Sep 17 00:58:21.503525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503443    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44672\" (UniqueName: \"kubernetes.io/projected/e4110dd9-03ae-4f71-b4bc-bb08c9951629-kube-api-access-44672\") pod \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\" (UID: \"e4110dd9-03ae-4f71-b4bc-bb08c9951629\") "
Sep 17 00:58:21.503525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503490    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966a4e59-4cc2-4eb9-a0ea-70f855272550-tls-certs\") pod \"966a4e59-4cc2-4eb9-a0ea-70f855272550\" (UID: \"966a4e59-4cc2-4eb9-a0ea-70f855272550\") "
Sep 17 00:58:21.503525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.503516    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-dshm\") pod \"2d2469d0-da52-435b-bc40-43a27fc06195\" (UID: \"2d2469d0-da52-435b-bc40-43a27fc06195\") "
Sep 17 00:58:21.504467 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.504183    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-home" (OuterVolumeSpecName: "home") pod "966a4e59-4cc2-4eb9-a0ea-70f855272550" (UID: "966a4e59-4cc2-4eb9-a0ea-70f855272550"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.504650 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.504561    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4110dd9-03ae-4f71-b4bc-bb08c9951629" (UID: "e4110dd9-03ae-4f71-b4bc-bb08c9951629"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.504650 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.504585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "966a4e59-4cc2-4eb9-a0ea-70f855272550" (UID: "966a4e59-4cc2-4eb9-a0ea-70f855272550"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.504919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.504823    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.504919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.504852    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.504919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.504866    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.504919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.504879    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.504919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.504891    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.504919 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.504902    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.505495 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.505464    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-dshm" (OuterVolumeSpecName: "dshm") pod "966a4e59-4cc2-4eb9-a0ea-70f855272550" (UID: "966a4e59-4cc2-4eb9-a0ea-70f855272550"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.506593 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.506554    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-dshm" (OuterVolumeSpecName: "dshm") pod "e4110dd9-03ae-4f71-b4bc-bb08c9951629" (UID: "e4110dd9-03ae-4f71-b4bc-bb08c9951629"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.506822 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.506712    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2469d0-da52-435b-bc40-43a27fc06195-kube-api-access-bpsz4" (OuterVolumeSpecName: "kube-api-access-bpsz4") pod "2d2469d0-da52-435b-bc40-43a27fc06195" (UID: "2d2469d0-da52-435b-bc40-43a27fc06195"). InnerVolumeSpecName "kube-api-access-bpsz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:58:21.506883 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.506866    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2469d0-da52-435b-bc40-43a27fc06195-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d2469d0-da52-435b-bc40-43a27fc06195" (UID: "2d2469d0-da52-435b-bc40-43a27fc06195"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:58:21.507094 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.507059    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4110dd9-03ae-4f71-b4bc-bb08c9951629-kube-api-access-44672" (OuterVolumeSpecName: "kube-api-access-44672") pod "e4110dd9-03ae-4f71-b4bc-bb08c9951629" (UID: "e4110dd9-03ae-4f71-b4bc-bb08c9951629"). InnerVolumeSpecName "kube-api-access-44672". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:58:21.507598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.507564    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4110dd9-03ae-4f71-b4bc-bb08c9951629" (UID: "e4110dd9-03ae-4f71-b4bc-bb08c9951629"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:58:21.507814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.507789    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966a4e59-4cc2-4eb9-a0ea-70f855272550-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "966a4e59-4cc2-4eb9-a0ea-70f855272550" (UID: "966a4e59-4cc2-4eb9-a0ea-70f855272550"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:58:21.508221 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.508190    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-dshm" (OuterVolumeSpecName: "dshm") pod "2d2469d0-da52-435b-bc40-43a27fc06195" (UID: "2d2469d0-da52-435b-bc40-43a27fc06195"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:58:21.508711 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.508689    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966a4e59-4cc2-4eb9-a0ea-70f855272550-kube-api-access-jpn9z" (OuterVolumeSpecName: "kube-api-access-jpn9z") pod "966a4e59-4cc2-4eb9-a0ea-70f855272550" (UID: "966a4e59-4cc2-4eb9-a0ea-70f855272550"). InnerVolumeSpecName "kube-api-access-jpn9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:58:21.606171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606071    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966a4e59-4cc2-4eb9-a0ea-70f855272550-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606096    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2469d0-da52-435b-bc40-43a27fc06195-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606106    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606114    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606122    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpsz4\" (UniqueName: \"kubernetes.io/projected/2d2469d0-da52-435b-bc40-43a27fc06195-kube-api-access-bpsz4\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606130    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2469d0-da52-435b-bc40-43a27fc06195-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606138    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4110dd9-03ae-4f71-b4bc-bb08c9951629-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606171 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606176    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606527 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606184    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4110dd9-03ae-4f71-b4bc-bb08c9951629-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606527 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606192    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpn9z\" (UniqueName: \"kubernetes.io/projected/966a4e59-4cc2-4eb9-a0ea-70f855272550-kube-api-access-jpn9z\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606527 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606199    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966a4e59-4cc2-4eb9-a0ea-70f855272550-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:21.606527 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:21.606210    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44672\" (UniqueName: \"kubernetes.io/projected/e4110dd9-03ae-4f71-b4bc-bb08c9951629-kube-api-access-44672\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:58:22.024813 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.024773    2576 generic.go:356] "Generic (PLEG): container finished" podID="2d2469d0-da52-435b-bc40-43a27fc06195" containerID="d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36" exitCode=0
Sep 17 00:58:22.025018 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.024831    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"
Sep 17 00:58:22.025018 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.024866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9" event={"ID":"2d2469d0-da52-435b-bc40-43a27fc06195","Type":"ContainerDied","Data":"d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36"}
Sep 17 00:58:22.025018 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.024915    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9" event={"ID":"2d2469d0-da52-435b-bc40-43a27fc06195","Type":"ContainerDied","Data":"8d50a26e0efe81f90b3ce7953b033d622d9cb2a7a2276b55703bb34e4c8b8240"}
Sep 17 00:58:22.025018 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.024937    2576 scope.go:117] "RemoveContainer" containerID="d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36"
Sep 17 00:58:22.026894 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.026855    2576 generic.go:356] "Generic (PLEG): container finished" podID="e4110dd9-03ae-4f71-b4bc-bb08c9951629" containerID="9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416" exitCode=0
Sep 17 00:58:22.027012 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.026914    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"
Sep 17 00:58:22.027012 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.026938    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt" event={"ID":"e4110dd9-03ae-4f71-b4bc-bb08c9951629","Type":"ContainerDied","Data":"9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416"}
Sep 17 00:58:22.027012 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.026982    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt" event={"ID":"e4110dd9-03ae-4f71-b4bc-bb08c9951629","Type":"ContainerDied","Data":"2ca3e0f6bb6202fbf85e2d999a65ae9583b2722fbdb026206893479a0e90baf5"}
Sep 17 00:58:22.028462 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.028441    2576 generic.go:356] "Generic (PLEG): container finished" podID="966a4e59-4cc2-4eb9-a0ea-70f855272550" containerID="c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21" exitCode=0
Sep 17 00:58:22.028573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.028480    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"
Sep 17 00:58:22.028573 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.028544    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp" event={"ID":"966a4e59-4cc2-4eb9-a0ea-70f855272550","Type":"ContainerDied","Data":"c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21"}
Sep 17 00:58:22.028672 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.028579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp" event={"ID":"966a4e59-4cc2-4eb9-a0ea-70f855272550","Type":"ContainerDied","Data":"534a3e2207d483538cf4146b1902fc045c403b63cb7e09417b93313fe99d9354"}
Sep 17 00:58:22.029019 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.028872    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:22.031461 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.031281    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:22.038173 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.038076    2576 scope.go:117] "RemoveContainer" containerID="d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36"
Sep 17 00:58:22.040897 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:58:22.040748    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36\": container with ID starting with d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36 not found: ID does not exist" containerID="d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36"
Sep 17 00:58:22.040897 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.040788    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36"} err="failed to get container status \"d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36\": rpc error: code = NotFound desc = could not find container \"d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36\": container with ID starting with d49df78b62cd6cc24ca640e49f30aebffdb8aa89fbefd44f9b1506409709ff36 not found: ID does not exist"
Sep 17 00:58:22.040897 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.040814    2576 scope.go:117] "RemoveContainer" containerID="9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416"
Sep 17 00:58:22.051506 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.051441    2576 scope.go:117] "RemoveContainer" containerID="9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416"
Sep 17 00:58:22.051863 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:58:22.051816    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416\": container with ID starting with 9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416 not found: ID does not exist" containerID="9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416"
Sep 17 00:58:22.051973 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.051917    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416"} err="failed to get container status \"9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416\": rpc error: code = NotFound desc = could not find container \"9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416\": container with ID starting with 9899da5e6f22867169810090ff4c629d6d4b8571db5f363b97327e817a27c416 not found: ID does not exist"
Sep 17 00:58:22.051973 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.051961    2576 scope.go:117] "RemoveContainer" containerID="c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21"
Sep 17 00:58:22.062890 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.062869    2576 scope.go:117] "RemoveContainer" containerID="c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21"
Sep 17 00:58:22.063284 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:58:22.063256    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21\": container with ID starting with c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21 not found: ID does not exist" containerID="c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21"
Sep 17 00:58:22.063384 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.063297    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21"} err="failed to get container status \"c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21\": rpc error: code = NotFound desc = could not find container \"c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21\": container with ID starting with c4b30bec4c582cfb87786256560be74e02290d97ad4e6f2d68134c2e3f30dd21 not found: ID does not exist"
Sep 17 00:58:22.065795 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.065768    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:22.067055 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.067035    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"]
Sep 17 00:58:22.073945 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.073916    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/distinct-w0-a79f56-kserve-79f685ff9f-h2dj9"]
Sep 17 00:58:22.084952 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.084912    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"]
Sep 17 00:58:22.087583 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.087561    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/embedding-w0-a79f56-kserve-7678689584-7n8jp"]
Sep 17 00:58:22.097845 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.097819    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"]
Sep 17 00:58:22.099674 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.099654    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/premium-w0-a79f56-kserve-5cc7c85756-b4szt"]
Sep 17 00:58:22.588303 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.588272    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d2469d0-da52-435b-bc40-43a27fc06195" path="/var/lib/kubelet/pods/2d2469d0-da52-435b-bc40-43a27fc06195/volumes"
Sep 17 00:58:22.588719 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.588703    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966a4e59-4cc2-4eb9-a0ea-70f855272550" path="/var/lib/kubelet/pods/966a4e59-4cc2-4eb9-a0ea-70f855272550/volumes"
Sep 17 00:58:22.589052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.589041    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4110dd9-03ae-4f71-b4bc-bb08c9951629" path="/var/lib/kubelet/pods/e4110dd9-03ae-4f71-b4bc-bb08c9951629/volumes"
Sep 17 00:58:22.808314 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.808284    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:22.909358 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:22.909265    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:23.013228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:23.013190    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:23.656489 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:23.656451    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:23.756887 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:23.756851    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:23.858213 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:23.858178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:23.957182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:23.957066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:24.527257 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:24.527225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:25.263368 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:25.263334    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:25.971770 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:25.971690    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:26.071778 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.071742    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:26.173351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.173311    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:26.402172 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.402125    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:26.500512 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.500483    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:26.507388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507362    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"]
Sep 17 00:58:26.507758 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507744    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2469d0-da52-435b-bc40-43a27fc06195" containerName="main"
Sep 17 00:58:26.507758 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507758    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2469d0-da52-435b-bc40-43a27fc06195" containerName="main"
Sep 17 00:58:26.507848 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507777    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966a4e59-4cc2-4eb9-a0ea-70f855272550" containerName="main"
Sep 17 00:58:26.507848 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507782    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="966a4e59-4cc2-4eb9-a0ea-70f855272550" containerName="main"
Sep 17 00:58:26.507848 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507792    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4110dd9-03ae-4f71-b4bc-bb08c9951629" containerName="main"
Sep 17 00:58:26.507848 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507798    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4110dd9-03ae-4f71-b4bc-bb08c9951629" containerName="main"
Sep 17 00:58:26.507987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507861    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2469d0-da52-435b-bc40-43a27fc06195" containerName="main"
Sep 17 00:58:26.507987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507869    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="966a4e59-4cc2-4eb9-a0ea-70f855272550" containerName="main"
Sep 17 00:58:26.507987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.507878    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4110dd9-03ae-4f71-b4bc-bb08c9951629" containerName="main"
Sep 17 00:58:26.512469 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.512452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:26.514867 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.514846    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-80e99a\""
Sep 17 00:58:26.514978 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.514962    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-80e99a-dockercfg-n4hml\""
Sep 17 00:58:26.522119 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.522099    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"]
Sep 17 00:58:26.540857 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.540826    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"]
Sep 17 00:58:26.545013 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.544993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:26.552382 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.552354    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"]
Sep 17 00:58:26.556924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.556895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5t6k\" (UniqueName: \"kubernetes.io/projected/e7f36d87-2417-4686-a394-327e5645ee07-kube-api-access-f5t6k\") pod \"payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx\" (UID: \"e7f36d87-2417-4686-a394-327e5645ee07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:26.557071 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.556996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f36d87-2417-4686-a394-327e5645ee07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx\" (UID: \"e7f36d87-2417-4686-a394-327e5645ee07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:26.657886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.657786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c06e5-f3bc-4139-b626-bb02d705e89f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt\" (UID: \"845c06e5-f3bc-4139-b626-bb02d705e89f\") " pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:26.657886 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.657836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f36d87-2417-4686-a394-327e5645ee07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx\" (UID: \"e7f36d87-2417-4686-a394-327e5645ee07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:26.658074 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.657881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqkg\" (UniqueName: \"kubernetes.io/projected/845c06e5-f3bc-4139-b626-bb02d705e89f-kube-api-access-clqkg\") pod \"payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt\" (UID: \"845c06e5-f3bc-4139-b626-bb02d705e89f\") " pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:26.658074 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.658061    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5t6k\" (UniqueName: \"kubernetes.io/projected/e7f36d87-2417-4686-a394-327e5645ee07-kube-api-access-f5t6k\") pod \"payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx\" (UID: \"e7f36d87-2417-4686-a394-327e5645ee07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:26.658509 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.658491    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f36d87-2417-4686-a394-327e5645ee07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx\" (UID: \"e7f36d87-2417-4686-a394-327e5645ee07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:26.665349 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.665323    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5t6k\" (UniqueName: \"kubernetes.io/projected/e7f36d87-2417-4686-a394-327e5645ee07-kube-api-access-f5t6k\") pod \"payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx\" (UID: \"e7f36d87-2417-4686-a394-327e5645ee07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:26.759589 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.759552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c06e5-f3bc-4139-b626-bb02d705e89f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt\" (UID: \"845c06e5-f3bc-4139-b626-bb02d705e89f\") " pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:26.759589 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.759599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqkg\" (UniqueName: \"kubernetes.io/projected/845c06e5-f3bc-4139-b626-bb02d705e89f-kube-api-access-clqkg\") pod \"payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt\" (UID: \"845c06e5-f3bc-4139-b626-bb02d705e89f\") " pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:26.760310 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.760290    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c06e5-f3bc-4139-b626-bb02d705e89f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt\" (UID: \"845c06e5-f3bc-4139-b626-bb02d705e89f\") " pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:26.767065 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.767040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqkg\" (UniqueName: \"kubernetes.io/projected/845c06e5-f3bc-4139-b626-bb02d705e89f-kube-api-access-clqkg\") pod \"payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt\" (UID: \"845c06e5-f3bc-4139-b626-bb02d705e89f\") " pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:26.823995 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.823957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:26.857550 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:26.857518    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:27.101048 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:27.101023    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"]
Sep 17 00:58:27.102530 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:58:27.102502    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f36d87_2417_4686_a394_327e5645ee07.slice/crio-7cf3231604ef7e71f23ba2fdbf9a84e594f25677b05de3ee3c7779c7131c49f1 WatchSource:0}: Error finding container 7cf3231604ef7e71f23ba2fdbf9a84e594f25677b05de3ee3c7779c7131c49f1: Status 404 returned error can't find the container with id 7cf3231604ef7e71f23ba2fdbf9a84e594f25677b05de3ee3c7779c7131c49f1
Sep 17 00:58:27.143330 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:27.143184    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"]
Sep 17 00:58:27.150374 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:58:27.150342    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845c06e5_f3bc_4139_b626_bb02d705e89f.slice/crio-6f5d1d00e51c9a764db9c54a053e3048c729b6599395e6d96fd7993d557d97e8 WatchSource:0}: Error finding container 6f5d1d00e51c9a764db9c54a053e3048c729b6599395e6d96fd7993d557d97e8: Status 404 returned error can't find the container with id 6f5d1d00e51c9a764db9c54a053e3048c729b6599395e6d96fd7993d557d97e8
Sep 17 00:58:28.068937 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:28.068898    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx" event={"ID":"e7f36d87-2417-4686-a394-327e5645ee07","Type":"ContainerStarted","Data":"a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185"}
Sep 17 00:58:28.069397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:28.069003    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:28.069397 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:28.069030    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx" event={"ID":"e7f36d87-2417-4686-a394-327e5645ee07","Type":"ContainerStarted","Data":"7cf3231604ef7e71f23ba2fdbf9a84e594f25677b05de3ee3c7779c7131c49f1"}
Sep 17 00:58:28.071038 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:28.071009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt" event={"ID":"845c06e5-f3bc-4139-b626-bb02d705e89f","Type":"ContainerStarted","Data":"f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6"}
Sep 17 00:58:28.071188 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:28.071055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt" event={"ID":"845c06e5-f3bc-4139-b626-bb02d705e89f","Type":"ContainerStarted","Data":"6f5d1d00e51c9a764db9c54a053e3048c729b6599395e6d96fd7993d557d97e8"}
Sep 17 00:58:28.071188 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:28.071104    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:28.085577 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:28.085530    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx" podStartSLOduration=2.085512729 podStartE2EDuration="2.085512729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:58:28.084178711 +0000 UTC m=+1704.039775051" watchObservedRunningTime="2026-09-17 00:58:28.085512729 +0000 UTC m=+1704.041109074"
Sep 17 00:58:28.098455 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:28.098405    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt" podStartSLOduration=2.098390239 podStartE2EDuration="2.098390239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:58:28.097767234 +0000 UTC m=+1704.053363578" watchObservedRunningTime="2026-09-17 00:58:28.098390239 +0000 UTC m=+1704.053986579"
Sep 17 00:58:39.079388 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:39.079344    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:58:39.079893 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:39.079631    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:58:40.181446 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:40.181417    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:43.131018 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:43.130981    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:43.232098 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:43.232065    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:43.348195 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:43.348170    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:44.169103 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:44.169074    2576 generic.go:356] "Generic (PLEG): container finished" podID="2356fc31-e2c5-4309-9742-62d620edd243" containerID="72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8" exitCode=0
Sep 17 00:58:44.169482 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:44.169114    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" event={"ID":"2356fc31-e2c5-4309-9742-62d620edd243","Type":"ContainerDied","Data":"72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8"}
Sep 17 00:58:44.169526 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:44.169494    2576 scope.go:117] "RemoveContainer" containerID="72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8"
Sep 17 00:58:44.170555 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:44.170497    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:58:44.170617 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:44.170609    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:58:44.170653 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:44.170642    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:58:45.178168 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:45.178116    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" event={"ID":"2356fc31-e2c5-4309-9742-62d620edd243","Type":"ContainerStarted","Data":"78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257"}
Sep 17 00:58:46.070331 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:46.070289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:46.070524 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:46.070482    2576 patch_prober.go:28] interesting pod/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 17 00:58:46.070625 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:46.070552    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 17 00:58:46.376962 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:46.376884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:47.058657 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:47.058618    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:47.070875 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:47.070842    2576 patch_prober.go:28] interesting pod/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 17 00:58:47.071026 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:47.070907    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 17 00:58:48.070810 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:48.070781    2576 patch_prober.go:28] interesting pod/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 17 00:58:48.071182 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:48.070841    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 17 00:58:48.246504 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:48.246471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:48.939965 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:48.939922    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:49.070258 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:49.070219    2576 patch_prober.go:28] interesting pod/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 17 00:58:49.070446 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:49.070286    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 17 00:58:49.754070 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:49.754016    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:50.070326 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:50.070286    2576 patch_prober.go:28] interesting pod/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 17 00:58:50.070506 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:50.070348    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 17 00:58:50.396696 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:50.396569    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:51.055428 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:51.055399    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:51.074451 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:51.074427    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:51.074843 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:51.074818    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:51.075434 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:51.075415    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:58:51.770957 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:51.770927    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:52.965411 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:52.965379    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:53.402551 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.402522    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:53.663987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.663891    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"]
Sep 17 00:58:53.668462 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.668436    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.670799 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.670777    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1c4c89-openshift-default-dockercfg-mhdpf\""
Sep 17 00:58:53.677822 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.677795    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"]
Sep 17 00:58:53.729538 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.729514    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:53.826075 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.826041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwmd\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-kube-api-access-vrwmd\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.826305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.826093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-istio-token\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.826305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.826220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab31233d-6825-4ba0-a81d-f71604305e48-istiod-ca-cert\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.826305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.826246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-credential-socket\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.826305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.826273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-envoy\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.826305 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.826297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-socket\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.826507 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.826371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-certs\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.826507 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.826458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-data\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.826507 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.826497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab31233d-6825-4ba0-a81d-f71604305e48-istio-podinfo\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.927777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.927701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab31233d-6825-4ba0-a81d-f71604305e48-istiod-ca-cert\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.927777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.927729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-credential-socket\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.927777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.927747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-envoy\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.927777 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.927763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-socket\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928058 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.927792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-certs\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928058 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.927850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-data\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928058 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.927888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab31233d-6825-4ba0-a81d-f71604305e48-istio-podinfo\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928058 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.927929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwmd\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-kube-api-access-vrwmd\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.928162    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-credential-socket\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.928248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-istio-token\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.928253    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-data\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928629 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.928604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-socket\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928770 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.928746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-certs\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.928891 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.928876    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab31233d-6825-4ba0-a81d-f71604305e48-istiod-ca-cert\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.930999 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.930978    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-envoy\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.931174 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.931130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab31233d-6825-4ba0-a81d-f71604305e48-istio-podinfo\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.935939 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.935915    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-istio-token\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.936099 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.936082    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwmd\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-kube-api-access-vrwmd\") pod \"e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") " pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:53.980044 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:53.980018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:54.256841 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:54.256710    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"]
Sep 17 00:58:54.259897 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:58:54.259870    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab31233d_6825_4ba0_a81d_f71604305e48.slice/crio-a7f6a2edf37f8c3fb5139072f36c87b0b31db21b5021e40d9245c03a1e8eba2b WatchSource:0}: Error finding container a7f6a2edf37f8c3fb5139072f36c87b0b31db21b5021e40d9245c03a1e8eba2b: Status 404 returned error can't find the container with id a7f6a2edf37f8c3fb5139072f36c87b0b31db21b5021e40d9245c03a1e8eba2b
Sep 17 00:58:54.262491 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:54.262458    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:58:54.262620 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:54.262534    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:58:54.262620 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:54.262586    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:58:54.333162 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:54.333124    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:55.237106 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:55.237069    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" event={"ID":"ab31233d-6825-4ba0-a81d-f71604305e48","Type":"ContainerStarted","Data":"4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673"}
Sep 17 00:58:55.237591 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:55.237137    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" event={"ID":"ab31233d-6825-4ba0-a81d-f71604305e48","Type":"ContainerStarted","Data":"a7f6a2edf37f8c3fb5139072f36c87b0b31db21b5021e40d9245c03a1e8eba2b"}
Sep 17 00:58:55.258058 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:55.258011    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" podStartSLOduration=2.257996585 podStartE2EDuration="2.257996585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:58:55.256915178 +0000 UTC m=+1731.212511519" watchObservedRunningTime="2026-09-17 00:58:55.257996585 +0000 UTC m=+1731.213592926"
Sep 17 00:58:55.980844 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:55.980814    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:55.985538 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:55.985516    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:56.243120 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:56.243042    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:56.244075 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:56.244054    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:58:56.553525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:56.553486    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:57.156052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:57.156025    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:57.254908 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:57.254875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:58.073816 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:58.073790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:58.431663 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:58.431588    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:58.813497 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:58.813459    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:58:59.550716 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:58:59.550686    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:00.185012 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:00.184984    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:00.830096 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:00.830069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:00.935427 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:00.935401    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:01.265738 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:01.265658    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:01.368522 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:01.368488    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:01.690371 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:01.690341    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:04.079465 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:04.079427    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:04.853384 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:04.853348    2576 scope.go:117] "RemoveContainer" containerID="5036c035968ad86b0d6d1d9a20828bfa67bc7388178da220599c0eb5fcaaf18c"
Sep 17 00:59:04.862485 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:04.862467    2576 scope.go:117] "RemoveContainer" containerID="e1121e6885e6ab7a9c386ed1947d0298b9784d3a3ca300bdd0e7750b5ac93052"
Sep 17 00:59:15.964201 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:15.964163    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:21.724669 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:21.724637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:22.409939 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:22.409901    2576 generic.go:356] "Generic (PLEG): container finished" podID="ab31233d-6825-4ba0-a81d-f71604305e48" containerID="4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673" exitCode=0
Sep 17 00:59:22.410113 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:22.409961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" event={"ID":"ab31233d-6825-4ba0-a81d-f71604305e48","Type":"ContainerDied","Data":"4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673"}
Sep 17 00:59:22.410419 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:22.410404    2576 scope.go:117] "RemoveContainer" containerID="4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673"
Sep 17 00:59:22.411601 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:22.411569    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:59:22.411657 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:22.411641    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:59:22.411692 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:22.411671    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 00:59:23.152869 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:23.152832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:23.418032 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:23.417948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" event={"ID":"ab31233d-6825-4ba0-a81d-f71604305e48","Type":"ContainerStarted","Data":"b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361"}
Sep 17 00:59:23.886470 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:23.886429    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:23.980246 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:23.980206    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:59:23.981422 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:23.981397    2576 patch_prober.go:28] interesting pod/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 17 00:59:23.981539 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:23.981456    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 17 00:59:24.654961 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:24.654929    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:24.981123 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:24.981035    2576 patch_prober.go:28] interesting pod/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 17 00:59:24.981123 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:24.981089    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 17 00:59:25.270691 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:25.270609    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:25.981217 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:25.981183    2576 patch_prober.go:28] interesting pod/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 17 00:59:25.981638 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:25.981240    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 17 00:59:26.049636 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:26.049607    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:26.243796 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:26.243697    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:59:26.686928 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:26.686894    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:26.980668 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:26.980583    2576 patch_prober.go:28] interesting pod/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 17 00:59:26.980668 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:26.980634    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 17 00:59:27.213003 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:27.212972    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:27.317340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:27.317294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:27.980717 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:27.980682    2576 patch_prober.go:28] interesting pod/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 17 00:59:27.980904 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:27.980767    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 17 00:59:28.985340 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:28.985312    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:59:28.986434 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:28.986412    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 00:59:29.790672 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:29.790639    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:30.060991 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.060906    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"]
Sep 17 00:59:30.065525 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.065510    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"
Sep 17 00:59:30.068106 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.068077    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:59:30.072758 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.072736    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"]
Sep 17 00:59:30.197925 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.197883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e755882-2f63-4d6c-be7d-c5d81ddc9dad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv\" (UID: \"2e755882-2f63-4d6c-be7d-c5d81ddc9dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"
Sep 17 00:59:30.298856 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.298820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e755882-2f63-4d6c-be7d-c5d81ddc9dad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv\" (UID: \"2e755882-2f63-4d6c-be7d-c5d81ddc9dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"
Sep 17 00:59:30.299482 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.299464    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e755882-2f63-4d6c-be7d-c5d81ddc9dad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv\" (UID: \"2e755882-2f63-4d6c-be7d-c5d81ddc9dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"
Sep 17 00:59:30.379761 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.379701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"
Sep 17 00:59:30.584042 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.584012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:30.656052 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:30.656027    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"]
Sep 17 00:59:30.658223 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:59:30.658129    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e755882_2f63_4d6c_be7d_c5d81ddc9dad.slice/crio-206767b93049e7efb21ab98221448c1aade580a9a84be13cd4062dd70548b505 WatchSource:0}: Error finding container 206767b93049e7efb21ab98221448c1aade580a9a84be13cd4062dd70548b505: Status 404 returned error can't find the container with id 206767b93049e7efb21ab98221448c1aade580a9a84be13cd4062dd70548b505
Sep 17 00:59:31.346255 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:31.346206    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:31.470236 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:31.470205    2576 generic.go:356] "Generic (PLEG): container finished" podID="2e755882-2f63-4d6c-be7d-c5d81ddc9dad" containerID="47f3001e0ea33e269e94abdde01d627b019b8a6985cc33d9abc1cd44d6466c7b" exitCode=0
Sep 17 00:59:31.470429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:31.470292    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv" event={"ID":"2e755882-2f63-4d6c-be7d-c5d81ddc9dad","Type":"ContainerDied","Data":"47f3001e0ea33e269e94abdde01d627b019b8a6985cc33d9abc1cd44d6466c7b"}
Sep 17 00:59:31.470429 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:31.470365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv" event={"ID":"2e755882-2f63-4d6c-be7d-c5d81ddc9dad","Type":"ContainerStarted","Data":"206767b93049e7efb21ab98221448c1aade580a9a84be13cd4062dd70548b505"}
Sep 17 00:59:32.053251 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:32.053217    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:32.615946 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:32.615925    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"
Sep 17 00:59:32.722981 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:32.722954    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e755882-2f63-4d6c-be7d-c5d81ddc9dad-maas-api-service-ca\") pod \"2e755882-2f63-4d6c-be7d-c5d81ddc9dad\" (UID: \"2e755882-2f63-4d6c-be7d-c5d81ddc9dad\") "
Sep 17 00:59:32.723344 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:32.723320    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e755882-2f63-4d6c-be7d-c5d81ddc9dad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e755882-2f63-4d6c-be7d-c5d81ddc9dad" (UID: "2e755882-2f63-4d6c-be7d-c5d81ddc9dad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:59:32.795111 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:32.795069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:32.824456 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:32.824435    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e755882-2f63-4d6c-be7d-c5d81ddc9dad-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:33.481947 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:33.481919    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"
Sep 17 00:59:33.482121 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:33.481919    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv" event={"ID":"2e755882-2f63-4d6c-be7d-c5d81ddc9dad","Type":"ContainerDied","Data":"206767b93049e7efb21ab98221448c1aade580a9a84be13cd4062dd70548b505"}
Sep 17 00:59:33.482121 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:33.482038    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="206767b93049e7efb21ab98221448c1aade580a9a84be13cd4062dd70548b505"
Sep 17 00:59:33.493508 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:33.493487    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:34.298224 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:34.298197    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:35.123181 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:35.123128    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:40.194413 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:40.194383    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:45.464484 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.464453    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:45.756716 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.756641    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:45.784175 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.784132    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"]
Sep 17 00:59:45.784395 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.784374    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt" podUID="845c06e5-f3bc-4139-b626-bb02d705e89f" containerName="payload-processing" containerID="cri-o://f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6" gracePeriod=30
Sep 17 00:59:45.789117 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.789093    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"]
Sep 17 00:59:45.789502 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.789478    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx" podUID="e7f36d87-2417-4686-a394-327e5645ee07" containerName="payload-pre-processing" containerID="cri-o://a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185" gracePeriod=30
Sep 17 00:59:45.822733 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.822695    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:45.881166 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.881120    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:45.980572 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.980538    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:45.988491 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.988466    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"]
Sep 17 00:59:45.991665 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:45.991645    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-80e99a-bmjhv"]
Sep 17 00:59:46.046311 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.046291    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:59:46.065910 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.065753    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:46.076747 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.076727    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:59:46.139865 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.139831    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f36d87-2417-4686-a394-327e5645ee07-plugins-config-volume\") pod \"e7f36d87-2417-4686-a394-327e5645ee07\" (UID: \"e7f36d87-2417-4686-a394-327e5645ee07\") "
Sep 17 00:59:46.140060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.139906    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clqkg\" (UniqueName: \"kubernetes.io/projected/845c06e5-f3bc-4139-b626-bb02d705e89f-kube-api-access-clqkg\") pod \"845c06e5-f3bc-4139-b626-bb02d705e89f\" (UID: \"845c06e5-f3bc-4139-b626-bb02d705e89f\") "
Sep 17 00:59:46.140060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.139957    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5t6k\" (UniqueName: \"kubernetes.io/projected/e7f36d87-2417-4686-a394-327e5645ee07-kube-api-access-f5t6k\") pod \"e7f36d87-2417-4686-a394-327e5645ee07\" (UID: \"e7f36d87-2417-4686-a394-327e5645ee07\") "
Sep 17 00:59:46.140060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.140043    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c06e5-f3bc-4139-b626-bb02d705e89f-plugins-config-volume\") pod \"845c06e5-f3bc-4139-b626-bb02d705e89f\" (UID: \"845c06e5-f3bc-4139-b626-bb02d705e89f\") "
Sep 17 00:59:46.140262 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.140244    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f36d87-2417-4686-a394-327e5645ee07-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7f36d87-2417-4686-a394-327e5645ee07" (UID: "e7f36d87-2417-4686-a394-327e5645ee07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:59:46.140598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.140571    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f36d87-2417-4686-a394-327e5645ee07-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:46.140598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.140572    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845c06e5-f3bc-4139-b626-bb02d705e89f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "845c06e5-f3bc-4139-b626-bb02d705e89f" (UID: "845c06e5-f3bc-4139-b626-bb02d705e89f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:59:46.143216 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.143183    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f36d87-2417-4686-a394-327e5645ee07-kube-api-access-f5t6k" (OuterVolumeSpecName: "kube-api-access-f5t6k") pod "e7f36d87-2417-4686-a394-327e5645ee07" (UID: "e7f36d87-2417-4686-a394-327e5645ee07"). InnerVolumeSpecName "kube-api-access-f5t6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:59:46.143674 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.143643    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845c06e5-f3bc-4139-b626-bb02d705e89f-kube-api-access-clqkg" (OuterVolumeSpecName: "kube-api-access-clqkg") pod "845c06e5-f3bc-4139-b626-bb02d705e89f" (UID: "845c06e5-f3bc-4139-b626-bb02d705e89f"). InnerVolumeSpecName "kube-api-access-clqkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:59:46.241491 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.241460    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5t6k\" (UniqueName: \"kubernetes.io/projected/e7f36d87-2417-4686-a394-327e5645ee07-kube-api-access-f5t6k\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:46.241491 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.241488    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c06e5-f3bc-4139-b626-bb02d705e89f-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:46.241491 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.241499    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clqkg\" (UniqueName: \"kubernetes.io/projected/845c06e5-f3bc-4139-b626-bb02d705e89f-kube-api-access-clqkg\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:46.338055 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.338010    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:46.438228 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.438197    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:46.554521 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.554492    2576 generic.go:356] "Generic (PLEG): container finished" podID="e7f36d87-2417-4686-a394-327e5645ee07" containerID="a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185" exitCode=0
Sep 17 00:59:46.554967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.554564    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"
Sep 17 00:59:46.554967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.554602    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx" event={"ID":"e7f36d87-2417-4686-a394-327e5645ee07","Type":"ContainerDied","Data":"a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185"}
Sep 17 00:59:46.554967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.554675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx" event={"ID":"e7f36d87-2417-4686-a394-327e5645ee07","Type":"ContainerDied","Data":"7cf3231604ef7e71f23ba2fdbf9a84e594f25677b05de3ee3c7779c7131c49f1"}
Sep 17 00:59:46.554967 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.554709    2576 scope.go:117] "RemoveContainer" containerID="a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185"
Sep 17 00:59:46.556060 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.556037    2576 generic.go:356] "Generic (PLEG): container finished" podID="845c06e5-f3bc-4139-b626-bb02d705e89f" containerID="f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6" exitCode=0
Sep 17 00:59:46.556164 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.556074    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"
Sep 17 00:59:46.556164 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.556117    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt" event={"ID":"845c06e5-f3bc-4139-b626-bb02d705e89f","Type":"ContainerDied","Data":"f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6"}
Sep 17 00:59:46.556275 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.556181    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt" event={"ID":"845c06e5-f3bc-4139-b626-bb02d705e89f","Type":"ContainerDied","Data":"6f5d1d00e51c9a764db9c54a053e3048c729b6599395e6d96fd7993d557d97e8"}
Sep 17 00:59:46.567281 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.567259    2576 scope.go:117] "RemoveContainer" containerID="a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185"
Sep 17 00:59:46.568680 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:59:46.568187    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185\": container with ID starting with a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185 not found: ID does not exist" containerID="a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185"
Sep 17 00:59:46.568680 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.568218    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185"} err="failed to get container status \"a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185\": rpc error: code = NotFound desc = could not find container \"a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185\": container with ID starting with a84ae579d5b7800dfcabe174e9c2abeb47e358d41f14e3f9d0e850b384ffd185 not found: ID does not exist"
Sep 17 00:59:46.568680 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.568260    2576 scope.go:117] "RemoveContainer" containerID="f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6"
Sep 17 00:59:46.591916 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.591893    2576 scope.go:117] "RemoveContainer" containerID="f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6"
Sep 17 00:59:46.592214 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:59:46.592192    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6\": container with ID starting with f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6 not found: ID does not exist" containerID="f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6"
Sep 17 00:59:46.592316 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.592223    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6"} err="failed to get container status \"f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6\": rpc error: code = NotFound desc = could not find container \"f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6\": container with ID starting with f2150786aafefc21fd0ba1192cd780d8eae6346fc1d473e2f090dd5c169118f6 not found: ID does not exist"
Sep 17 00:59:46.592710 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.592689    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e755882-2f63-4d6c-be7d-c5d81ddc9dad" path="/var/lib/kubelet/pods/2e755882-2f63-4d6c-be7d-c5d81ddc9dad/volumes"
Sep 17 00:59:46.592987 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.592974    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"]
Sep 17 00:59:46.593107 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.593095    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-80e99a-69dcf7d9d7-dfdpt"]
Sep 17 00:59:46.601323 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.601302    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"]
Sep 17 00:59:46.605046 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:46.605026    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-80e99a-7665745b6-lh5tx"]
Sep 17 00:59:47.104594 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:47.104529    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:47.671645 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:47.671614    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"]
Sep 17 00:59:47.693402 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:47.693376    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:47.759374 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:47.759347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:48.563451 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:48.563411    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:48.587632 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:48.587607    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845c06e5-f3bc-4139-b626-bb02d705e89f" path="/var/lib/kubelet/pods/845c06e5-f3bc-4139-b626-bb02d705e89f/volumes"
Sep 17 00:59:48.587974 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:48.587962    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f36d87-2417-4686-a394-327e5645ee07" path="/var/lib/kubelet/pods/e7f36d87-2417-4686-a394-327e5645ee07/volumes"
Sep 17 00:59:49.182979 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:49.182942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:49.819925 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:49.819897    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:52.386870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.386834    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:52.487241 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.487179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:52.604535 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.604501    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"]
Sep 17 00:59:52.604766 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.604735    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy" containerID="cri-o://78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257" gracePeriod=30
Sep 17 00:59:52.625594 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.625570    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:52.938450 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938418    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"]
Sep 17 00:59:52.938860 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938844    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845c06e5-f3bc-4139-b626-bb02d705e89f" containerName="payload-processing"
Sep 17 00:59:52.938943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938865    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="845c06e5-f3bc-4139-b626-bb02d705e89f" containerName="payload-processing"
Sep 17 00:59:52.938943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938888    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e755882-2f63-4d6c-be7d-c5d81ddc9dad" containerName="revoke-keys"
Sep 17 00:59:52.938943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938894    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e755882-2f63-4d6c-be7d-c5d81ddc9dad" containerName="revoke-keys"
Sep 17 00:59:52.938943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938909    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f36d87-2417-4686-a394-327e5645ee07" containerName="payload-pre-processing"
Sep 17 00:59:52.938943 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938914    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f36d87-2417-4686-a394-327e5645ee07" containerName="payload-pre-processing"
Sep 17 00:59:52.939224 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938978    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="845c06e5-f3bc-4139-b626-bb02d705e89f" containerName="payload-processing"
Sep 17 00:59:52.939224 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938985    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f36d87-2417-4686-a394-327e5645ee07" containerName="payload-pre-processing"
Sep 17 00:59:52.939224 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.938991    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e755882-2f63-4d6c-be7d-c5d81ddc9dad" containerName="revoke-keys"
Sep 17 00:59:52.943461 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.943439    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"
Sep 17 00:59:52.946413 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.945944    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 00:59:52.948461 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:52.948439    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"]
Sep 17 00:59:53.005152 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:53.005126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70045729-108f-4ae1-b5e7-f46e62c32cf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp\" (UID: \"70045729-108f-4ae1-b5e7-f46e62c32cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"
Sep 17 00:59:53.105908 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:53.105886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70045729-108f-4ae1-b5e7-f46e62c32cf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp\" (UID: \"70045729-108f-4ae1-b5e7-f46e62c32cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"
Sep 17 00:59:53.106505 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:53.106485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70045729-108f-4ae1-b5e7-f46e62c32cf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp\" (UID: \"70045729-108f-4ae1-b5e7-f46e62c32cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"
Sep 17 00:59:53.255205 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:53.255125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"
Sep 17 00:59:53.262870 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:53.262852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:53.531464 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:53.531442    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"]
Sep 17 00:59:53.533789 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:59:53.533763    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70045729_108f_4ae1_b5e7_f46e62c32cf1.slice/crio-2dc8d3a596f64660d2e1703defccee32350b236fe35812afea4043f396348b58 WatchSource:0}: Error finding container 2dc8d3a596f64660d2e1703defccee32350b236fe35812afea4043f396348b58: Status 404 returned error can't find the container with id 2dc8d3a596f64660d2e1703defccee32350b236fe35812afea4043f396348b58
Sep 17 00:59:53.601041 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:53.601013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp" event={"ID":"70045729-108f-4ae1-b5e7-f46e62c32cf1","Type":"ContainerStarted","Data":"2dc8d3a596f64660d2e1703defccee32350b236fe35812afea4043f396348b58"}
Sep 17 00:59:54.053248 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:54.053218    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:54.607416 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:54.607390    2576 generic.go:356] "Generic (PLEG): container finished" podID="70045729-108f-4ae1-b5e7-f46e62c32cf1" containerID="1882b9551a5486ae4acb81fd4b9231457ee5683506a4b131126180cb39f6a77f" exitCode=0
Sep 17 00:59:54.607763 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:54.607451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp" event={"ID":"70045729-108f-4ae1-b5e7-f46e62c32cf1","Type":"ContainerDied","Data":"1882b9551a5486ae4acb81fd4b9231457ee5683506a4b131126180cb39f6a77f"}
Sep 17 00:59:54.674849 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:54.674822    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:55.293642 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:55.293599    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:55.754920 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:55.754899    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"
Sep 17 00:59:55.831580 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:55.831554    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70045729-108f-4ae1-b5e7-f46e62c32cf1-maas-api-service-ca\") pod \"70045729-108f-4ae1-b5e7-f46e62c32cf1\" (UID: \"70045729-108f-4ae1-b5e7-f46e62c32cf1\") "
Sep 17 00:59:55.831877 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:55.831853    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70045729-108f-4ae1-b5e7-f46e62c32cf1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70045729-108f-4ae1-b5e7-f46e62c32cf1" (UID: "70045729-108f-4ae1-b5e7-f46e62c32cf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:59:55.831948 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:55.831933    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70045729-108f-4ae1-b5e7-f46e62c32cf1-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:55.931268 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:55.931196    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:56.619536 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:56.619484    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp" event={"ID":"70045729-108f-4ae1-b5e7-f46e62c32cf1","Type":"ContainerDied","Data":"2dc8d3a596f64660d2e1703defccee32350b236fe35812afea4043f396348b58"}
Sep 17 00:59:56.619711 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:56.619545    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dc8d3a596f64660d2e1703defccee32350b236fe35812afea4043f396348b58"
Sep 17 00:59:56.619711 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:56.619613    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"
Sep 17 00:59:56.651247 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:56.651219    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:57.455699 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.455664    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:57.863546 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.863526    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:59:57.952452 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952424    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-istio-token\") pod \"2356fc31-e2c5-4309-9742-62d620edd243\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") "
Sep 17 00:59:57.952598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2356fc31-e2c5-4309-9742-62d620edd243-istio-podinfo\") pod \"2356fc31-e2c5-4309-9742-62d620edd243\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") "
Sep 17 00:59:57.952598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952487    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2356fc31-e2c5-4309-9742-62d620edd243-istiod-ca-cert\") pod \"2356fc31-e2c5-4309-9742-62d620edd243\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") "
Sep 17 00:59:57.952598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952537    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjdkq\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-kube-api-access-gjdkq\") pod \"2356fc31-e2c5-4309-9742-62d620edd243\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") "
Sep 17 00:59:57.952598 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952566    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-credential-socket\") pod \"2356fc31-e2c5-4309-9742-62d620edd243\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") "
Sep 17 00:59:57.952814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952611    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-envoy\") pod \"2356fc31-e2c5-4309-9742-62d620edd243\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") "
Sep 17 00:59:57.952814 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952638    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-certs\") pod \"2356fc31-e2c5-4309-9742-62d620edd243\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") "
Sep 17 00:59:57.952924 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952861    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2356fc31-e2c5-4309-9742-62d620edd243" (UID: "2356fc31-e2c5-4309-9742-62d620edd243"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:59:57.952983 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952945    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2356fc31-e2c5-4309-9742-62d620edd243" (UID: "2356fc31-e2c5-4309-9742-62d620edd243"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:59:57.953037 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952999    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-data\") pod \"2356fc31-e2c5-4309-9742-62d620edd243\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") "
Sep 17 00:59:57.953100 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.952994    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2356fc31-e2c5-4309-9742-62d620edd243-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2356fc31-e2c5-4309-9742-62d620edd243" (UID: "2356fc31-e2c5-4309-9742-62d620edd243"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:59:57.953100 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.953050    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-socket\") pod \"2356fc31-e2c5-4309-9742-62d620edd243\" (UID: \"2356fc31-e2c5-4309-9742-62d620edd243\") "
Sep 17 00:59:57.953404 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.953375    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-data" (OuterVolumeSpecName: "istio-data") pod "2356fc31-e2c5-4309-9742-62d620edd243" (UID: "2356fc31-e2c5-4309-9742-62d620edd243"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:59:57.953404 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.953391    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2356fc31-e2c5-4309-9742-62d620edd243" (UID: "2356fc31-e2c5-4309-9742-62d620edd243"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:59:57.953680 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.953660    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2356fc31-e2c5-4309-9742-62d620edd243-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:57.953757 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.953687    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:57.953757 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.953704    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:57.953757 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.953719    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:57.953757 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.953733    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:57.954933 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.954908    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2356fc31-e2c5-4309-9742-62d620edd243-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2356fc31-e2c5-4309-9742-62d620edd243" (UID: "2356fc31-e2c5-4309-9742-62d620edd243"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 00:59:57.955036 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.954965    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2356fc31-e2c5-4309-9742-62d620edd243" (UID: "2356fc31-e2c5-4309-9742-62d620edd243"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:59:57.955220 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.955205    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-istio-token" (OuterVolumeSpecName: "istio-token") pod "2356fc31-e2c5-4309-9742-62d620edd243" (UID: "2356fc31-e2c5-4309-9742-62d620edd243"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:59:57.955284 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:57.955239    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-kube-api-access-gjdkq" (OuterVolumeSpecName: "kube-api-access-gjdkq") pod "2356fc31-e2c5-4309-9742-62d620edd243" (UID: "2356fc31-e2c5-4309-9742-62d620edd243"). InnerVolumeSpecName "kube-api-access-gjdkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:59:58.054116 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.054089    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdkq\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-kube-api-access-gjdkq\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:58.054116 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.054113    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2356fc31-e2c5-4309-9742-62d620edd243-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:58.054267 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.054127    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2356fc31-e2c5-4309-9742-62d620edd243-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:58.054267 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.054138    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2356fc31-e2c5-4309-9742-62d620edd243-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 00:59:58.632449 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.632419    2576 generic.go:356] "Generic (PLEG): container finished" podID="2356fc31-e2c5-4309-9742-62d620edd243" containerID="78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257" exitCode=0
Sep 17 00:59:58.632817 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.632470    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"
Sep 17 00:59:58.632817 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.632500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" event={"ID":"2356fc31-e2c5-4309-9742-62d620edd243","Type":"ContainerDied","Data":"78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257"}
Sep 17 00:59:58.632817 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.632549    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h" event={"ID":"2356fc31-e2c5-4309-9742-62d620edd243","Type":"ContainerDied","Data":"cd82d0099b4a0e357def989715e88227cc1ebfcc59ff20d36dbd56095c8756d9"}
Sep 17 00:59:58.632817 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.632570    2576 scope.go:117] "RemoveContainer" containerID="78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257"
Sep 17 00:59:58.642007 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.641991    2576 scope.go:117] "RemoveContainer" containerID="72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8"
Sep 17 00:59:58.649392 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.649370    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"]
Sep 17 00:59:58.650607 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.650559    2576 scope.go:117] "RemoveContainer" containerID="78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257"
Sep 17 00:59:58.651069 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:59:58.651042    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257\": container with ID starting with 78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257 not found: ID does not exist" containerID="78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257"
Sep 17 00:59:58.651191 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.651079    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257"} err="failed to get container status \"78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257\": rpc error: code = NotFound desc = could not find container \"78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257\": container with ID starting with 78f4f456ad348305d5105287450e5e5e233d33f6272c353463c7dec7010e5257 not found: ID does not exist"
Sep 17 00:59:58.651191 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.651103    2576 scope.go:117] "RemoveContainer" containerID="72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8"
Sep 17 00:59:58.651464 ip-10-0-143-51 kubenswrapper[2576]: E0917 00:59:58.651443    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8\": container with ID starting with 72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8 not found: ID does not exist" containerID="72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8"
Sep 17 00:59:58.651540 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.651474    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8"} err="failed to get container status \"72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8\": rpc error: code = NotFound desc = could not find container \"72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8\": container with ID starting with 72f3c2e9389b2e8bcf7e8f712ba0977c2fb18fb74ea01a2294f8cde8cdeb08c8 not found: ID does not exist"
Sep 17 00:59:58.653351 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:58.653332    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-80e99a-openshift-default-5744f8656-p2m6h"]
Sep 17 00:59:59.098209 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098180    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"]
Sep 17 00:59:59.098604 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098588    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy"
Sep 17 00:59:59.098604 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098603    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy"
Sep 17 00:59:59.098736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098627    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70045729-108f-4ae1-b5e7-f46e62c32cf1" containerName="revoke-keys"
Sep 17 00:59:59.098736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098636    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="70045729-108f-4ae1-b5e7-f46e62c32cf1" containerName="revoke-keys"
Sep 17 00:59:59.098736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098649    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy"
Sep 17 00:59:59.098736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098655    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy"
Sep 17 00:59:59.098736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098717    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy"
Sep 17 00:59:59.098736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098724    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2356fc31-e2c5-4309-9742-62d620edd243" containerName="istio-proxy"
Sep 17 00:59:59.098736 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.098734    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="70045729-108f-4ae1-b5e7-f46e62c32cf1" containerName="revoke-keys"
Sep 17 00:59:59.103519 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.103495    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.106528 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.106291    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"kube-root-ca.crt\""
Sep 17 00:59:59.106528 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.106305    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"default-dockercfg-qjg2n\""
Sep 17 00:59:59.106528 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.106343    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"openshift-service-ca.crt\""
Sep 17 00:59:59.106528 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.106291    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"sub-test-model-d2326d-kserve-self-signed-certs\""
Sep 17 00:59:59.108941 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.108919    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"]
Sep 17 00:59:59.114780 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.114735    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:59.163004 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.162979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-dshm\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.163124 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.163008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-tmp-dir\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.163124 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.163056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8946bed9-87a6-41fa-95cf-59a2e1e44621-tls-certs\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.163124 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.163077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62hc\" (UniqueName: \"kubernetes.io/projected/8946bed9-87a6-41fa-95cf-59a2e1e44621-kube-api-access-d62hc\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.163124 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.163108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-home\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.163417 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.163192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-model-cache\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.264338 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.264305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-home\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.264469 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.264359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-model-cache\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.264469 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.264436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-dshm\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.264469 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.264463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-tmp-dir\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.264640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.264535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8946bed9-87a6-41fa-95cf-59a2e1e44621-tls-certs\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.264640 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.264565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62hc\" (UniqueName: \"kubernetes.io/projected/8946bed9-87a6-41fa-95cf-59a2e1e44621-kube-api-access-d62hc\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.264878 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.264827    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-home\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.265353 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.265327    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-model-cache\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.265481 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.265448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-tmp-dir\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.268599 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.268367    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-dshm\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.268704 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.268680    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8946bed9-87a6-41fa-95cf-59a2e1e44621-tls-certs\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.272317 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.272283    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62hc\" (UniqueName: \"kubernetes.io/projected/8946bed9-87a6-41fa-95cf-59a2e1e44621-kube-api-access-d62hc\") pod \"sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.414873 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.414804    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 00:59:59.688512 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.688487    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"]
Sep 17 00:59:59.691058 ip-10-0-143-51 kubenswrapper[2576]: W0917 00:59:59.691029    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8946bed9_87a6_41fa_95cf_59a2e1e44621.slice/crio-59175b7730de86dfb168117ff5f8ddc591e974330450813ea86893427a1d9744 WatchSource:0}: Error finding container 59175b7730de86dfb168117ff5f8ddc591e974330450813ea86893427a1d9744: Status 404 returned error can't find the container with id 59175b7730de86dfb168117ff5f8ddc591e974330450813ea86893427a1d9744
Sep 17 00:59:59.726641 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.726609    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 00:59:59.827254 ip-10-0-143-51 kubenswrapper[2576]: I0917 00:59:59.827230    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:00.135337 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.135311    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb"]
Sep 17 01:00:00.138886 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.138855    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2"]
Sep 17 01:00:00.139013 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.138971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb"
Sep 17 01:00:00.141674 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.141649    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 01:00:00.142660 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.142641    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj"]
Sep 17 01:00:00.142791 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.142780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2"
Sep 17 01:00:00.146598 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.146570    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj"
Sep 17 01:00:00.148054 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.147901    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb"]
Sep 17 01:00:00.151080 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.151062    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj"]
Sep 17 01:00:00.154336 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.154193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb"
Sep 17 01:00:00.154468 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.154454    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2"]
Sep 17 01:00:00.156839 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.156819    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx"]
Sep 17 01:00:00.161117 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.161098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx"
Sep 17 01:00:00.166262 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.166241    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2"
Sep 17 01:00:00.166602 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.166573    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx"]
Sep 17 01:00:00.179221 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.179200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj"
Sep 17 01:00:00.189116 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.189081    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx"
Sep 17 01:00:00.451319 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.449225    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb"]
Sep 17 01:00:00.460823 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.460797    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2"]
Sep 17 01:00:00.462922 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:00:00.462898    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b8f5cb_df8e_4da8_a46b_6eadf4fb25a5.slice/crio-153c7c07d43ab258b7d3f46f2a463d0f24db0628815f07f1e56504c5e2d2ac31 WatchSource:0}: Error finding container 153c7c07d43ab258b7d3f46f2a463d0f24db0628815f07f1e56504c5e2d2ac31: Status 404 returned error can't find the container with id 153c7c07d43ab258b7d3f46f2a463d0f24db0628815f07f1e56504c5e2d2ac31
Sep 17 01:00:00.498664 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.498641    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj"]
Sep 17 01:00:00.500532 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.500513    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx"]
Sep 17 01:00:00.500841 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:00:00.500824    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb44e21f_cae7_4edf_8d27_b4398c8ee8ed.slice/crio-f884c2407cc059fb62fe947fd70b0067bceed67e14fb60419eab75d3557084c1 WatchSource:0}: Error finding container f884c2407cc059fb62fe947fd70b0067bceed67e14fb60419eab75d3557084c1: Status 404 returned error can't find the container with id f884c2407cc059fb62fe947fd70b0067bceed67e14fb60419eab75d3557084c1
Sep 17 01:00:00.502235 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:00:00.502213    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1689fc9b_0eaa_4cae_9d44_f1f9e22654e6.slice/crio-0891e4e501df43ab2aba9cb2c881f7530dc1f1b094568700fe552998c4d58c75 WatchSource:0}: Error finding container 0891e4e501df43ab2aba9cb2c881f7530dc1f1b094568700fe552998c4d58c75: Status 404 returned error can't find the container with id 0891e4e501df43ab2aba9cb2c881f7530dc1f1b094568700fe552998c4d58c75
Sep 17 01:00:00.587504 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.587473    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2356fc31-e2c5-4309-9742-62d620edd243" path="/var/lib/kubelet/pods/2356fc31-e2c5-4309-9742-62d620edd243/volumes"
Sep 17 01:00:00.647450 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.647422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb" event={"ID":"a98d8393-d4db-4241-ac6c-b9c32e3724a3","Type":"ContainerStarted","Data":"3c870dbab078b4c12fadeb3399fc564f49c158e587711640e27d477848cbe8c2"}
Sep 17 01:00:00.648730 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.648703    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj" event={"ID":"bb44e21f-cae7-4edf-8d27-b4398c8ee8ed","Type":"ContainerStarted","Data":"f884c2407cc059fb62fe947fd70b0067bceed67e14fb60419eab75d3557084c1"}
Sep 17 01:00:00.649912 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.649886    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx" event={"ID":"1689fc9b-0eaa-4cae-9d44-f1f9e22654e6","Type":"ContainerStarted","Data":"0891e4e501df43ab2aba9cb2c881f7530dc1f1b094568700fe552998c4d58c75"}
Sep 17 01:00:00.651027 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.651004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2" event={"ID":"39b8f5cb-df8e-4da8-a46b-6eadf4fb25a5","Type":"ContainerStarted","Data":"153c7c07d43ab258b7d3f46f2a463d0f24db0628815f07f1e56504c5e2d2ac31"}
Sep 17 01:00:00.652521 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.652499    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt" event={"ID":"8946bed9-87a6-41fa-95cf-59a2e1e44621","Type":"ContainerStarted","Data":"e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844"}
Sep 17 01:00:00.652621 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.652530    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt" event={"ID":"8946bed9-87a6-41fa-95cf-59a2e1e44621","Type":"ContainerStarted","Data":"59175b7730de86dfb168117ff5f8ddc591e974330450813ea86893427a1d9744"}
Sep 17 01:00:00.668508 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:00.668464    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt" podStartSLOduration=1.668449275 podStartE2EDuration="1.668449275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:00:00.666971942 +0000 UTC m=+1796.622568283" watchObservedRunningTime="2026-09-17 01:00:00.668449275 +0000 UTC m=+1796.624045615"
Sep 17 01:00:01.659315 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:01.659291    2576 generic.go:356] "Generic (PLEG): container finished" podID="a98d8393-d4db-4241-ac6c-b9c32e3724a3" containerID="de724c6744fb05ee1b5a70a15426b31e08d0cb836ef916fc7e78824c13c90d35" exitCode=0
Sep 17 01:00:01.659664 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:01.659400    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb" event={"ID":"a98d8393-d4db-4241-ac6c-b9c32e3724a3","Type":"ContainerDied","Data":"de724c6744fb05ee1b5a70a15426b31e08d0cb836ef916fc7e78824c13c90d35"}
Sep 17 01:00:01.661007 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:01.660988    2576 generic.go:356] "Generic (PLEG): container finished" podID="bb44e21f-cae7-4edf-8d27-b4398c8ee8ed" containerID="4151f0c9f93a5946e5fcfd29dfc797b0524856aa05b46a4acccc0eef99f7d78d" exitCode=0
Sep 17 01:00:01.661087 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:01.661056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj" event={"ID":"bb44e21f-cae7-4edf-8d27-b4398c8ee8ed","Type":"ContainerDied","Data":"4151f0c9f93a5946e5fcfd29dfc797b0524856aa05b46a4acccc0eef99f7d78d"}
Sep 17 01:00:01.662463 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:01.662445    2576 generic.go:356] "Generic (PLEG): container finished" podID="1689fc9b-0eaa-4cae-9d44-f1f9e22654e6" containerID="dd138b061fde240400b153818934758bdc5035b9963dbd162ae67195ab200ea3" exitCode=0
Sep 17 01:00:01.662542 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:01.662501    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx" event={"ID":"1689fc9b-0eaa-4cae-9d44-f1f9e22654e6","Type":"ContainerDied","Data":"dd138b061fde240400b153818934758bdc5035b9963dbd162ae67195ab200ea3"}
Sep 17 01:00:01.663925 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:01.663906    2576 generic.go:356] "Generic (PLEG): container finished" podID="39b8f5cb-df8e-4da8-a46b-6eadf4fb25a5" containerID="04e544e5195e34a4295b6226f430d55e1fbdb14b365dae15d654fa43edf89be9" exitCode=0
Sep 17 01:00:01.663992 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:01.663977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2" event={"ID":"39b8f5cb-df8e-4da8-a46b-6eadf4fb25a5","Type":"ContainerDied","Data":"04e544e5195e34a4295b6226f430d55e1fbdb14b365dae15d654fa43edf89be9"}
Sep 17 01:00:02.692536 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:02.692455    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp" podUID="e3d04bf9-836a-4732-a9b0-ca04648e61a9" containerName="main" containerID="cri-o://9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413" gracePeriod=45
Sep 17 01:00:02.842066 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:02.841929    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb"
Sep 17 01:00:02.939724 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:02.939698    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2"
Sep 17 01:00:02.941909 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:02.941887    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj"
Sep 17 01:00:02.945426 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:02.945405    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx"
Sep 17 01:00:02.981360 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:02.981335    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 01:00:03.100277 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100247    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tmp-dir\") pod \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") "
Sep 17 01:00:03.100277 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100280    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-dshm\") pod \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") "
Sep 17 01:00:03.100506 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100320    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-home\") pod \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") "
Sep 17 01:00:03.100506 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100411    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tls-certs\") pod \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") "
Sep 17 01:00:03.100506 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100466    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw28c\" (UniqueName: \"kubernetes.io/projected/e3d04bf9-836a-4732-a9b0-ca04648e61a9-kube-api-access-sw28c\") pod \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") "
Sep 17 01:00:03.100658 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100501    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3d04bf9-836a-4732-a9b0-ca04648e61a9" (UID: "e3d04bf9-836a-4732-a9b0-ca04648e61a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:00:03.100658 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100514    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-model-cache\") pod \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\" (UID: \"e3d04bf9-836a-4732-a9b0-ca04648e61a9\") "
Sep 17 01:00:03.100658 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100647    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-home" (OuterVolumeSpecName: "home") pod "e3d04bf9-836a-4732-a9b0-ca04648e61a9" (UID: "e3d04bf9-836a-4732-a9b0-ca04648e61a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:00:03.100801 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100716    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3d04bf9-836a-4732-a9b0-ca04648e61a9" (UID: "e3d04bf9-836a-4732-a9b0-ca04648e61a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:00:03.100920 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100900    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:03.100920 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100921    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:03.101048 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.100930    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:03.102545 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.102522    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-dshm" (OuterVolumeSpecName: "dshm") pod "e3d04bf9-836a-4732-a9b0-ca04648e61a9" (UID: "e3d04bf9-836a-4732-a9b0-ca04648e61a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:00:03.102644 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.102595    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d04bf9-836a-4732-a9b0-ca04648e61a9-kube-api-access-sw28c" (OuterVolumeSpecName: "kube-api-access-sw28c") pod "e3d04bf9-836a-4732-a9b0-ca04648e61a9" (UID: "e3d04bf9-836a-4732-a9b0-ca04648e61a9"). InnerVolumeSpecName "kube-api-access-sw28c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:00:03.102644 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.102614    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3d04bf9-836a-4732-a9b0-ca04648e61a9" (UID: "e3d04bf9-836a-4732-a9b0-ca04648e61a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:00:03.201980 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.201922    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d04bf9-836a-4732-a9b0-ca04648e61a9-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:03.201980 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.201944    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d04bf9-836a-4732-a9b0-ca04648e61a9-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:03.201980 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.201953    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw28c\" (UniqueName: \"kubernetes.io/projected/e3d04bf9-836a-4732-a9b0-ca04648e61a9-kube-api-access-sw28c\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:03.679891 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.679867    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb"
Sep 17 01:00:03.680044 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.679913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826780-hwvzb" event={"ID":"a98d8393-d4db-4241-ac6c-b9c32e3724a3","Type":"ContainerDied","Data":"3c870dbab078b4c12fadeb3399fc564f49c158e587711640e27d477848cbe8c2"}
Sep 17 01:00:03.680044 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.679970    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c870dbab078b4c12fadeb3399fc564f49c158e587711640e27d477848cbe8c2"
Sep 17 01:00:03.681213 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.681193    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj"
Sep 17 01:00:03.681348 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.681200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj" event={"ID":"bb44e21f-cae7-4edf-8d27-b4398c8ee8ed","Type":"ContainerDied","Data":"f884c2407cc059fb62fe947fd70b0067bceed67e14fb60419eab75d3557084c1"}
Sep 17 01:00:03.681348 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.681305    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f884c2407cc059fb62fe947fd70b0067bceed67e14fb60419eab75d3557084c1"
Sep 17 01:00:03.682532 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.682510    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx"
Sep 17 01:00:03.682532 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.682526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx" event={"ID":"1689fc9b-0eaa-4cae-9d44-f1f9e22654e6","Type":"ContainerDied","Data":"0891e4e501df43ab2aba9cb2c881f7530dc1f1b094568700fe552998c4d58c75"}
Sep 17 01:00:03.682728 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.682561    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0891e4e501df43ab2aba9cb2c881f7530dc1f1b094568700fe552998c4d58c75"
Sep 17 01:00:03.683980 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.683949    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2"
Sep 17 01:00:03.683980 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.683957    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2" event={"ID":"39b8f5cb-df8e-4da8-a46b-6eadf4fb25a5","Type":"ContainerDied","Data":"153c7c07d43ab258b7d3f46f2a463d0f24db0628815f07f1e56504c5e2d2ac31"}
Sep 17 01:00:03.684205 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.683992    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="153c7c07d43ab258b7d3f46f2a463d0f24db0628815f07f1e56504c5e2d2ac31"
Sep 17 01:00:03.686242 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.686218    2576 generic.go:356] "Generic (PLEG): container finished" podID="e3d04bf9-836a-4732-a9b0-ca04648e61a9" containerID="9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413" exitCode=0
Sep 17 01:00:03.686376 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.686258    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"
Sep 17 01:00:03.686376 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.686312    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp" event={"ID":"e3d04bf9-836a-4732-a9b0-ca04648e61a9","Type":"ContainerDied","Data":"9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413"}
Sep 17 01:00:03.686376 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.686349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp" event={"ID":"e3d04bf9-836a-4732-a9b0-ca04648e61a9","Type":"ContainerDied","Data":"8787792da12c51971158d0a8bb3504665a62fc9d870332624907a4a59bef3d94"}
Sep 17 01:00:03.686376 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.686371    2576 scope.go:117] "RemoveContainer" containerID="9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413"
Sep 17 01:00:03.705186 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.705135    2576 scope.go:117] "RemoveContainer" containerID="9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413"
Sep 17 01:00:03.705587 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:00:03.705562    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413\": container with ID starting with 9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413 not found: ID does not exist" containerID="9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413"
Sep 17 01:00:03.705638 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.705600    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413"} err="failed to get container status \"9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413\": rpc error: code = NotFound desc = could not find container \"9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413\": container with ID starting with 9aa837cd8077a320084861492d58295dc93bac08c428e544f9139e70eaa14413 not found: ID does not exist"
Sep 17 01:00:03.726541 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.726515    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"]
Sep 17 01:00:03.731058 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:03.731037    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/auth-test-model-bcd112-kserve-664477f86c-qqtlp"]
Sep 17 01:00:04.588503 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:04.588473    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d04bf9-836a-4732-a9b0-ca04648e61a9" path="/var/lib/kubelet/pods/e3d04bf9-836a-4732-a9b0-ca04648e61a9/volumes"
Sep 17 01:00:05.090461 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:05.090431    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 01:00:05.093586 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:05.093566    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 01:00:05.906825 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:05.906782    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:06.163779 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:06.163704    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:08.404070 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:08.404037    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:08.539342 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:08.539302    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:08.626370 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:08.626345    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:08.679336 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:08.679260    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"]
Sep 17 01:00:08.682125 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:08.682103    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4c89-lkvfp"]
Sep 17 01:00:08.683689 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:08.683667    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:09.032139 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:09.032086    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:09.169083 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:09.169050    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:09.415570 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:09.415493    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 01:00:09.415570 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:09.415543    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 01:00:09.428720 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:09.428699    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 01:00:09.731775 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:09.731697    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 01:00:09.821161 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:09.821113    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:10.588117 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:10.588080    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70045729-108f-4ae1-b5e7-f46e62c32cf1" path="/var/lib/kubelet/pods/70045729-108f-4ae1-b5e7-f46e62c32cf1/volumes"
Sep 17 01:00:12.174676 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:12.174642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:14.868858 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:14.868824    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:14.969774 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:14.969728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:15.102691 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.102658    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"]
Sep 17 01:00:15.102883 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.102855    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy" containerID="cri-o://b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361" gracePeriod=30
Sep 17 01:00:15.121183 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.121104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:15.449685 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.449601    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"]
Sep 17 01:00:15.450090 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450072    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1689fc9b-0eaa-4cae-9d44-f1f9e22654e6" containerName="cleanup"
Sep 17 01:00:15.450199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450092    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1689fc9b-0eaa-4cae-9d44-f1f9e22654e6" containerName="cleanup"
Sep 17 01:00:15.450199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450105    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98d8393-d4db-4241-ac6c-b9c32e3724a3" containerName="cleanup"
Sep 17 01:00:15.450199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450112    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98d8393-d4db-4241-ac6c-b9c32e3724a3" containerName="cleanup"
Sep 17 01:00:15.450199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450123    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb44e21f-cae7-4edf-8d27-b4398c8ee8ed" containerName="cleanup"
Sep 17 01:00:15.450199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450131    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb44e21f-cae7-4edf-8d27-b4398c8ee8ed" containerName="cleanup"
Sep 17 01:00:15.450199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450171    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d04bf9-836a-4732-a9b0-ca04648e61a9" containerName="main"
Sep 17 01:00:15.450199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450180    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d04bf9-836a-4732-a9b0-ca04648e61a9" containerName="main"
Sep 17 01:00:15.450538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450206    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b8f5cb-df8e-4da8-a46b-6eadf4fb25a5" containerName="cleanup"
Sep 17 01:00:15.450538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450216    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b8f5cb-df8e-4da8-a46b-6eadf4fb25a5" containerName="cleanup"
Sep 17 01:00:15.450538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450324    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1689fc9b-0eaa-4cae-9d44-f1f9e22654e6" containerName="cleanup"
Sep 17 01:00:15.450538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450340    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d04bf9-836a-4732-a9b0-ca04648e61a9" containerName="main"
Sep 17 01:00:15.450538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450352    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb44e21f-cae7-4edf-8d27-b4398c8ee8ed" containerName="cleanup"
Sep 17 01:00:15.450538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450361    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b8f5cb-df8e-4da8-a46b-6eadf4fb25a5" containerName="cleanup"
Sep 17 01:00:15.450538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.450370    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98d8393-d4db-4241-ac6c-b9c32e3724a3" containerName="cleanup"
Sep 17 01:00:15.453833 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.453813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"
Sep 17 01:00:15.456665 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.456635    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 01:00:15.458937 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.458912    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"]
Sep 17 01:00:15.562380 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.562356    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:15.608801 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.608765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386afd84-15aa-4691-aef7-0ecec4497b24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q\" (UID: \"386afd84-15aa-4691-aef7-0ecec4497b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"
Sep 17 01:00:15.709639 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.709534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386afd84-15aa-4691-aef7-0ecec4497b24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q\" (UID: \"386afd84-15aa-4691-aef7-0ecec4497b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"
Sep 17 01:00:15.710240 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.710220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386afd84-15aa-4691-aef7-0ecec4497b24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q\" (UID: \"386afd84-15aa-4691-aef7-0ecec4497b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"
Sep 17 01:00:15.767261 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:15.767234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"
Sep 17 01:00:16.039018 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:16.038971    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"]
Sep 17 01:00:16.040203 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:00:16.040171    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386afd84_15aa_4691_aef7_0ecec4497b24.slice/crio-da4eb3062e46da51150527d625577c16a9835c221da2021ed5074ae99d4de041 WatchSource:0}: Error finding container da4eb3062e46da51150527d625577c16a9835c221da2021ed5074ae99d4de041: Status 404 returned error can't find the container with id da4eb3062e46da51150527d625577c16a9835c221da2021ed5074ae99d4de041
Sep 17 01:00:16.404203 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:16.404176    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:16.767606 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:16.767581    2576 generic.go:356] "Generic (PLEG): container finished" podID="386afd84-15aa-4691-aef7-0ecec4497b24" containerID="5a6edc125eae9df4eff5fad8aaed54ac5bf4caa3b83a0de4263494777d6904f0" exitCode=0
Sep 17 01:00:16.767728 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:16.767665    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q" event={"ID":"386afd84-15aa-4691-aef7-0ecec4497b24","Type":"ContainerDied","Data":"5a6edc125eae9df4eff5fad8aaed54ac5bf4caa3b83a0de4263494777d6904f0"}
Sep 17 01:00:16.767728 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:16.767716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q" event={"ID":"386afd84-15aa-4691-aef7-0ecec4497b24","Type":"ContainerStarted","Data":"da4eb3062e46da51150527d625577c16a9835c221da2021ed5074ae99d4de041"}
Sep 17 01:00:17.174078 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:17.174047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:17.744197 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:17.744162    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:17.918905 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:17.918882    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"
Sep 17 01:00:18.030420 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:18.030397    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386afd84-15aa-4691-aef7-0ecec4497b24-maas-api-service-ca\") pod \"386afd84-15aa-4691-aef7-0ecec4497b24\" (UID: \"386afd84-15aa-4691-aef7-0ecec4497b24\") "
Sep 17 01:00:18.030733 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:18.030712    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386afd84-15aa-4691-aef7-0ecec4497b24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "386afd84-15aa-4691-aef7-0ecec4497b24" (UID: "386afd84-15aa-4691-aef7-0ecec4497b24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:00:18.131484 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:18.131455    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386afd84-15aa-4691-aef7-0ecec4497b24-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:18.359375 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:18.359296    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:18.780109 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:18.780084    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"
Sep 17 01:00:18.780256 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:18.780120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q" event={"ID":"386afd84-15aa-4691-aef7-0ecec4497b24","Type":"ContainerDied","Data":"da4eb3062e46da51150527d625577c16a9835c221da2021ed5074ae99d4de041"}
Sep 17 01:00:18.780256 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:18.780186    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da4eb3062e46da51150527d625577c16a9835c221da2021ed5074ae99d4de041"
Sep 17 01:00:19.038337 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.038251    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:19.489542 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.489444    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"]
Sep 17 01:00:19.489969 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.489921    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386afd84-15aa-4691-aef7-0ecec4497b24" containerName="revoke-keys"
Sep 17 01:00:19.489969 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.489935    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="386afd84-15aa-4691-aef7-0ecec4497b24" containerName="revoke-keys"
Sep 17 01:00:19.490045 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.490015    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="386afd84-15aa-4691-aef7-0ecec4497b24" containerName="revoke-keys"
Sep 17 01:00:19.493059 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.493042    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.496615 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.496593    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"openshift-service-ca.crt\""
Sep 17 01:00:19.496710 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.496612    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"kube-root-ca.crt\""
Sep 17 01:00:19.496710 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.496669    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"sub-test-model-bcd112-kserve-self-signed-certs\""
Sep 17 01:00:19.496710 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.496593    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"default-dockercfg-qxzgf\""
Sep 17 01:00:19.503099 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.503080    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"]
Sep 17 01:00:19.512821 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.512795    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:19.646508 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.646482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-model-cache\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.646623 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.646531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfc553c-210d-4c52-8839-c1650284dbd5-tls-certs\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.646678 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.646616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-tmp-dir\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.646731 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.646717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-dshm\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.646770 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.646742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g94rw\" (UniqueName: \"kubernetes.io/projected/5cfc553c-210d-4c52-8839-c1650284dbd5-kube-api-access-g94rw\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.646770 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.646759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-home\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.748104 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.748027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfc553c-210d-4c52-8839-c1650284dbd5-tls-certs\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.748104 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.748068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-tmp-dir\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.748341 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.748190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-dshm\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.748341 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.748222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g94rw\" (UniqueName: \"kubernetes.io/projected/5cfc553c-210d-4c52-8839-c1650284dbd5-kube-api-access-g94rw\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.748341 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.748249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-home\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.748341 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.748294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-model-cache\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.748580 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.748555    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-tmp-dir\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.748680 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.748661    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-model-cache\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.748753 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.748727    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-home\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.750540 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.750514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-dshm\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.750713 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.750696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfc553c-210d-4c52-8839-c1650284dbd5-tls-certs\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.755703 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.755687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g94rw\" (UniqueName: \"kubernetes.io/projected/5cfc553c-210d-4c52-8839-c1650284dbd5-kube-api-access-g94rw\") pod \"sub-test-model-bcd112-kserve-59f644f597-hn949\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:19.803443 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:19.803421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:20.080007 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.079978    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"]
Sep 17 01:00:20.080778 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:00:20.080755    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cfc553c_210d_4c52_8839_c1650284dbd5.slice/crio-a73323e93592244559c28ae16a52591bfeedcfeddb2a827987987a45581ed0e4 WatchSource:0}: Error finding container a73323e93592244559c28ae16a52591bfeedcfeddb2a827987987a45581ed0e4: Status 404 returned error can't find the container with id a73323e93592244559c28ae16a52591bfeedcfeddb2a827987987a45581ed0e4
Sep 17 01:00:20.350565 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.350537    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 01:00:20.454862 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.454832    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrwmd\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-kube-api-access-vrwmd\") pod \"ab31233d-6825-4ba0-a81d-f71604305e48\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") "
Sep 17 01:00:20.455014 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.454874    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-credential-socket\") pod \"ab31233d-6825-4ba0-a81d-f71604305e48\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") "
Sep 17 01:00:20.455014 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.454909    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-istio-token\") pod \"ab31233d-6825-4ba0-a81d-f71604305e48\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") "
Sep 17 01:00:20.455014 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.454932    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-certs\") pod \"ab31233d-6825-4ba0-a81d-f71604305e48\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") "
Sep 17 01:00:20.455014 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.454977    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab31233d-6825-4ba0-a81d-f71604305e48-istiod-ca-cert\") pod \"ab31233d-6825-4ba0-a81d-f71604305e48\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") "
Sep 17 01:00:20.455014 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.454995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab31233d-6825-4ba0-a81d-f71604305e48-istio-podinfo\") pod \"ab31233d-6825-4ba0-a81d-f71604305e48\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") "
Sep 17 01:00:20.455337 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455079    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-envoy\") pod \"ab31233d-6825-4ba0-a81d-f71604305e48\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") "
Sep 17 01:00:20.455337 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455113    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-socket\") pod \"ab31233d-6825-4ba0-a81d-f71604305e48\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") "
Sep 17 01:00:20.455337 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455166    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-data\") pod \"ab31233d-6825-4ba0-a81d-f71604305e48\" (UID: \"ab31233d-6825-4ba0-a81d-f71604305e48\") "
Sep 17 01:00:20.455337 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455138    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ab31233d-6825-4ba0-a81d-f71604305e48" (UID: "ab31233d-6825-4ba0-a81d-f71604305e48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:00:20.455337 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455243    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ab31233d-6825-4ba0-a81d-f71604305e48" (UID: "ab31233d-6825-4ba0-a81d-f71604305e48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:00:20.455730 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455355    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ab31233d-6825-4ba0-a81d-f71604305e48" (UID: "ab31233d-6825-4ba0-a81d-f71604305e48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:00:20.455730 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab31233d-6825-4ba0-a81d-f71604305e48-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ab31233d-6825-4ba0-a81d-f71604305e48" (UID: "ab31233d-6825-4ba0-a81d-f71604305e48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:00:20.455730 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455527    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:20.455730 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455531    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-data" (OuterVolumeSpecName: "istio-data") pod "ab31233d-6825-4ba0-a81d-f71604305e48" (UID: "ab31233d-6825-4ba0-a81d-f71604305e48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:00:20.455730 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455548    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab31233d-6825-4ba0-a81d-f71604305e48-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:20.455730 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455561    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:20.455730 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.455574    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:20.457299 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.457273    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-kube-api-access-vrwmd" (OuterVolumeSpecName: "kube-api-access-vrwmd") pod "ab31233d-6825-4ba0-a81d-f71604305e48" (UID: "ab31233d-6825-4ba0-a81d-f71604305e48"). InnerVolumeSpecName "kube-api-access-vrwmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:00:20.457426 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.457323    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab31233d-6825-4ba0-a81d-f71604305e48" (UID: "ab31233d-6825-4ba0-a81d-f71604305e48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:00:20.457426 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.457357    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab31233d-6825-4ba0-a81d-f71604305e48-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ab31233d-6825-4ba0-a81d-f71604305e48" (UID: "ab31233d-6825-4ba0-a81d-f71604305e48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 01:00:20.457499 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.457436    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ab31233d-6825-4ba0-a81d-f71604305e48" (UID: "ab31233d-6825-4ba0-a81d-f71604305e48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:00:20.556621 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.556593    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:20.556621 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.556615    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab31233d-6825-4ba0-a81d-f71604305e48-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:20.556621 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.556624    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrwmd\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-kube-api-access-vrwmd\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:20.557002 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.556634    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab31233d-6825-4ba0-a81d-f71604305e48-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:20.557002 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.556644    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab31233d-6825-4ba0-a81d-f71604305e48-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:20.792330 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.792303    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949" event={"ID":"5cfc553c-210d-4c52-8839-c1650284dbd5","Type":"ContainerStarted","Data":"f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89"}
Sep 17 01:00:20.792472 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.792345    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949" event={"ID":"5cfc553c-210d-4c52-8839-c1650284dbd5","Type":"ContainerStarted","Data":"a73323e93592244559c28ae16a52591bfeedcfeddb2a827987987a45581ed0e4"}
Sep 17 01:00:20.794020 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.793995    2576 generic.go:356] "Generic (PLEG): container finished" podID="ab31233d-6825-4ba0-a81d-f71604305e48" containerID="b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361" exitCode=0
Sep 17 01:00:20.794115 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.794039    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"
Sep 17 01:00:20.794115 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.794060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" event={"ID":"ab31233d-6825-4ba0-a81d-f71604305e48","Type":"ContainerDied","Data":"b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361"}
Sep 17 01:00:20.794115 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.794104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn" event={"ID":"ab31233d-6825-4ba0-a81d-f71604305e48","Type":"ContainerDied","Data":"a7f6a2edf37f8c3fb5139072f36c87b0b31db21b5021e40d9245c03a1e8eba2b"}
Sep 17 01:00:20.794235 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.794133    2576 scope.go:117] "RemoveContainer" containerID="b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361"
Sep 17 01:00:20.803210 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.803194    2576 scope.go:117] "RemoveContainer" containerID="4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673"
Sep 17 01:00:20.810074 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.810035    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949" podStartSLOduration=1.810022217 podStartE2EDuration="1.810022217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:00:20.80787695 +0000 UTC m=+1816.763473291" watchObservedRunningTime="2026-09-17 01:00:20.810022217 +0000 UTC m=+1816.765618558"
Sep 17 01:00:20.812620 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.812454    2576 scope.go:117] "RemoveContainer" containerID="b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361"
Sep 17 01:00:20.812750 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:00:20.812732    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361\": container with ID starting with b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361 not found: ID does not exist" containerID="b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361"
Sep 17 01:00:20.812798 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.812757    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361"} err="failed to get container status \"b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361\": rpc error: code = NotFound desc = could not find container \"b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361\": container with ID starting with b3d02b6fdcd696b01367e2eeecee71ba3d77547792a4caff8cd4d098dda67361 not found: ID does not exist"
Sep 17 01:00:20.812798 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.812774    2576 scope.go:117] "RemoveContainer" containerID="4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673"
Sep 17 01:00:20.813031 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:00:20.813013    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673\": container with ID starting with 4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673 not found: ID does not exist" containerID="4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673"
Sep 17 01:00:20.813083 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.813039    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673"} err="failed to get container status \"4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673\": rpc error: code = NotFound desc = could not find container \"4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673\": container with ID starting with 4fdfd2e69dc3916e8405d8ff5b9bf8f1d6302f04228cb6cd69aa0651d359f673 not found: ID does not exist"
Sep 17 01:00:20.823023 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.822998    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"]
Sep 17 01:00:20.826369 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:20.826348    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1c4c89-openshift-default-747574d96b-d9hzn"]
Sep 17 01:00:22.588307 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:22.588273    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" path="/var/lib/kubelet/pods/ab31233d-6825-4ba0-a81d-f71604305e48/volumes"
Sep 17 01:00:25.886332 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:25.886293    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx"]
Sep 17 01:00:25.888516 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:25.888496    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-929f3a-29826780-bz5qx"]
Sep 17 01:00:26.107867 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:26.107831    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"]
Sep 17 01:00:26.112325 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:26.112300    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-929f3a-qdz5q"]
Sep 17 01:00:26.364060 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:26.364030    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:26.563471 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:26.563441    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:26.587382 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:26.587357    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1689fc9b-0eaa-4cae-9d44-f1f9e22654e6" path="/var/lib/kubelet/pods/1689fc9b-0eaa-4cae-9d44-f1f9e22654e6/volumes"
Sep 17 01:00:26.587677 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:26.587664    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386afd84-15aa-4691-aef7-0ecec4497b24" path="/var/lib/kubelet/pods/386afd84-15aa-4691-aef7-0ecec4497b24/volumes"
Sep 17 01:00:26.663982 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:26.663923    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:27.339826 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:27.339775    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:28.016660 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:28.016608    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:28.860714 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:28.860680    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:29.804483 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:29.804450    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:29.804660 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:29.804497    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:29.816762 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:29.816737    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:29.859354 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:29.859326    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:00:30.983079 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:30.983040    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:32.002617 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.002568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:32.103096 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.103027    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:32.240320 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.240290    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:32.564791 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.564759    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"]
Sep 17 01:00:32.565176 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.565162    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy"
Sep 17 01:00:32.565176 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.565177    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy"
Sep 17 01:00:32.565264 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.565191    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy"
Sep 17 01:00:32.565264 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.565198    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy"
Sep 17 01:00:32.565331 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.565267    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy"
Sep 17 01:00:32.565331 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.565275    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab31233d-6825-4ba0-a81d-f71604305e48" containerName="istio-proxy"
Sep 17 01:00:32.569991 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.569974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"
Sep 17 01:00:32.572741 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.572717    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 01:00:32.574192 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.574169    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"]
Sep 17 01:00:32.763853 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.763823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd2d31b-d8b6-49ed-b25e-6867b8e9c513-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td\" (UID: \"edd2d31b-d8b6-49ed-b25e-6867b8e9c513\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"
Sep 17 01:00:32.830823 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.830752    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:32.864759 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.864732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd2d31b-d8b6-49ed-b25e-6867b8e9c513-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td\" (UID: \"edd2d31b-d8b6-49ed-b25e-6867b8e9c513\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"
Sep 17 01:00:32.865307 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.865291    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd2d31b-d8b6-49ed-b25e-6867b8e9c513-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td\" (UID: \"edd2d31b-d8b6-49ed-b25e-6867b8e9c513\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"
Sep 17 01:00:32.882395 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:32.882378    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"
Sep 17 01:00:33.155493 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:33.155468    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"]
Sep 17 01:00:33.157720 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:00:33.157686    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd2d31b_d8b6_49ed_b25e_6867b8e9c513.slice/crio-2dd0310cd73d6be0281f818b7e4cf9cc5e65b49deb70b0a11010bbbdb2d88999 WatchSource:0}: Error finding container 2dd0310cd73d6be0281f818b7e4cf9cc5e65b49deb70b0a11010bbbdb2d88999: Status 404 returned error can't find the container with id 2dd0310cd73d6be0281f818b7e4cf9cc5e65b49deb70b0a11010bbbdb2d88999
Sep 17 01:00:33.526903 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:33.526874    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:33.871955 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:33.871930    2576 generic.go:356] "Generic (PLEG): container finished" podID="edd2d31b-d8b6-49ed-b25e-6867b8e9c513" containerID="99a89ae35b0ae894acb219440c55144bd9c3a3af04501760338a51058971b2c4" exitCode=0
Sep 17 01:00:33.872066 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:33.871992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td" event={"ID":"edd2d31b-d8b6-49ed-b25e-6867b8e9c513","Type":"ContainerDied","Data":"99a89ae35b0ae894acb219440c55144bd9c3a3af04501760338a51058971b2c4"}
Sep 17 01:00:33.872066 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:33.872032    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td" event={"ID":"edd2d31b-d8b6-49ed-b25e-6867b8e9c513","Type":"ContainerStarted","Data":"2dd0310cd73d6be0281f818b7e4cf9cc5e65b49deb70b0a11010bbbdb2d88999"}
Sep 17 01:00:34.157857 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:34.157785    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:34.776514 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:34.776480    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:35.003538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:35.003514    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"
Sep 17 01:00:35.085552 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:35.085478    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd2d31b-d8b6-49ed-b25e-6867b8e9c513-maas-api-service-ca\") pod \"edd2d31b-d8b6-49ed-b25e-6867b8e9c513\" (UID: \"edd2d31b-d8b6-49ed-b25e-6867b8e9c513\") "
Sep 17 01:00:35.085825 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:35.085806    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd2d31b-d8b6-49ed-b25e-6867b8e9c513-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edd2d31b-d8b6-49ed-b25e-6867b8e9c513" (UID: "edd2d31b-d8b6-49ed-b25e-6867b8e9c513"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:00:35.187020 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:35.186986    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd2d31b-d8b6-49ed-b25e-6867b8e9c513-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:35.646561 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:35.646528    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:35.884671 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:35.884646    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"
Sep 17 01:00:35.884824 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:35.884683    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td" event={"ID":"edd2d31b-d8b6-49ed-b25e-6867b8e9c513","Type":"ContainerDied","Data":"2dd0310cd73d6be0281f818b7e4cf9cc5e65b49deb70b0a11010bbbdb2d88999"}
Sep 17 01:00:35.884824 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:35.884727    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dd0310cd73d6be0281f818b7e4cf9cc5e65b49deb70b0a11010bbbdb2d88999"
Sep 17 01:00:43.019493 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.019456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:43.022384 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.022358    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj"]
Sep 17 01:00:43.025598 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.025574    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93017d-29826780-vxjwj"]
Sep 17 01:00:43.043237 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.042160    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"]
Sep 17 01:00:43.043237 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.042424    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m" podUID="310eff3a-f4ac-472e-bc99-1833c4e885a0" containerName="payload-processing" containerID="cri-o://e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd" gracePeriod=30
Sep 17 01:00:43.049814 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.049792    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"]
Sep 17 01:00:43.051210 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.051187    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv" podUID="33082d64-2f02-408e-b355-0d63dac4f89d" containerName="payload-pre-processing" containerID="cri-o://5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a" gracePeriod=30
Sep 17 01:00:43.083897 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.083872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:43.145135 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.145098    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:43.194655 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.194621    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"]
Sep 17 01:00:43.198763 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.198698    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93017d-242td"]
Sep 17 01:00:43.208626 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.208596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:43.309996 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.309974    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 01:00:43.332241 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.332210    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 01:00:43.352407 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.352377    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33082d64-2f02-408e-b355-0d63dac4f89d-plugins-config-volume\") pod \"33082d64-2f02-408e-b355-0d63dac4f89d\" (UID: \"33082d64-2f02-408e-b355-0d63dac4f89d\") "
Sep 17 01:00:43.352558 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.352417    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzjdb\" (UniqueName: \"kubernetes.io/projected/310eff3a-f4ac-472e-bc99-1833c4e885a0-kube-api-access-mzjdb\") pod \"310eff3a-f4ac-472e-bc99-1833c4e885a0\" (UID: \"310eff3a-f4ac-472e-bc99-1833c4e885a0\") "
Sep 17 01:00:43.352558 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.352454    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l5nx\" (UniqueName: \"kubernetes.io/projected/33082d64-2f02-408e-b355-0d63dac4f89d-kube-api-access-6l5nx\") pod \"33082d64-2f02-408e-b355-0d63dac4f89d\" (UID: \"33082d64-2f02-408e-b355-0d63dac4f89d\") "
Sep 17 01:00:43.352558 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.352490    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310eff3a-f4ac-472e-bc99-1833c4e885a0-plugins-config-volume\") pod \"310eff3a-f4ac-472e-bc99-1833c4e885a0\" (UID: \"310eff3a-f4ac-472e-bc99-1833c4e885a0\") "
Sep 17 01:00:43.352788 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.352765    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33082d64-2f02-408e-b355-0d63dac4f89d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33082d64-2f02-408e-b355-0d63dac4f89d" (UID: "33082d64-2f02-408e-b355-0d63dac4f89d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:00:43.352916 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.352888    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310eff3a-f4ac-472e-bc99-1833c4e885a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "310eff3a-f4ac-472e-bc99-1833c4e885a0" (UID: "310eff3a-f4ac-472e-bc99-1833c4e885a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:00:43.352916 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.352900    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33082d64-2f02-408e-b355-0d63dac4f89d-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:43.354752 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.354725    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310eff3a-f4ac-472e-bc99-1833c4e885a0-kube-api-access-mzjdb" (OuterVolumeSpecName: "kube-api-access-mzjdb") pod "310eff3a-f4ac-472e-bc99-1833c4e885a0" (UID: "310eff3a-f4ac-472e-bc99-1833c4e885a0"). InnerVolumeSpecName "kube-api-access-mzjdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:00:43.354853 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.354834    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33082d64-2f02-408e-b355-0d63dac4f89d-kube-api-access-6l5nx" (OuterVolumeSpecName: "kube-api-access-6l5nx") pod "33082d64-2f02-408e-b355-0d63dac4f89d" (UID: "33082d64-2f02-408e-b355-0d63dac4f89d"). InnerVolumeSpecName "kube-api-access-6l5nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:00:43.454012 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.453976    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzjdb\" (UniqueName: \"kubernetes.io/projected/310eff3a-f4ac-472e-bc99-1833c4e885a0-kube-api-access-mzjdb\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:43.454012 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.454004    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l5nx\" (UniqueName: \"kubernetes.io/projected/33082d64-2f02-408e-b355-0d63dac4f89d-kube-api-access-6l5nx\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:43.454012 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.454012    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310eff3a-f4ac-472e-bc99-1833c4e885a0-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:00:43.541574 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.541447    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:43.642458 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.642408    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:43.928939 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.928865    2576 generic.go:356] "Generic (PLEG): container finished" podID="310eff3a-f4ac-472e-bc99-1833c4e885a0" containerID="e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd" exitCode=0
Sep 17 01:00:43.928939 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.928908    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"
Sep 17 01:00:43.929127 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.928951    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m" event={"ID":"310eff3a-f4ac-472e-bc99-1833c4e885a0","Type":"ContainerDied","Data":"e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd"}
Sep 17 01:00:43.929127 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.929004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m" event={"ID":"310eff3a-f4ac-472e-bc99-1833c4e885a0","Type":"ContainerDied","Data":"8d665f69d92ca5f04fb577095ebf37cf8c96e9c8cc18eb7d18b902dba912f6b5"}
Sep 17 01:00:43.929127 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.929030    2576 scope.go:117] "RemoveContainer" containerID="e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd"
Sep 17 01:00:43.930444 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.930420    2576 generic.go:356] "Generic (PLEG): container finished" podID="33082d64-2f02-408e-b355-0d63dac4f89d" containerID="5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a" exitCode=0
Sep 17 01:00:43.930574 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.930471    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"
Sep 17 01:00:43.930574 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.930504    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv" event={"ID":"33082d64-2f02-408e-b355-0d63dac4f89d","Type":"ContainerDied","Data":"5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a"}
Sep 17 01:00:43.930574 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.930543    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv" event={"ID":"33082d64-2f02-408e-b355-0d63dac4f89d","Type":"ContainerDied","Data":"2c82e8a43047e672aa43bc27061b8807a79449bf884231909847963f760e8132"}
Sep 17 01:00:43.941392 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.941370    2576 scope.go:117] "RemoveContainer" containerID="e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd"
Sep 17 01:00:43.941862 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:00:43.941786    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd\": container with ID starting with e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd not found: ID does not exist" containerID="e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd"
Sep 17 01:00:43.941862 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.941823    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd"} err="failed to get container status \"e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd\": rpc error: code = NotFound desc = could not find container \"e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd\": container with ID starting with e4460fa87ebf0d27a5625e9e8bb1f7a2088452ccec23f28b1efd0b6077a8aefd not found: ID does not exist"
Sep 17 01:00:43.941862 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.941847    2576 scope.go:117] "RemoveContainer" containerID="5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a"
Sep 17 01:00:43.950742 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.950722    2576 scope.go:117] "RemoveContainer" containerID="5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a"
Sep 17 01:00:43.951039 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:00:43.951018    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a\": container with ID starting with 5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a not found: ID does not exist" containerID="5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a"
Sep 17 01:00:43.951109 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.951044    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a"} err="failed to get container status \"5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a\": rpc error: code = NotFound desc = could not find container \"5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a\": container with ID starting with 5c24fc76c6e9afe04a94c1588241c9e321869acf1f74084b6ea82c405753fc2a not found: ID does not exist"
Sep 17 01:00:43.951938 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.951920    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"]
Sep 17 01:00:43.958169 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.958149    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-93017d-757fb86cf9-f8d7m"]
Sep 17 01:00:43.968865 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.968847    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"]
Sep 17 01:00:43.970791 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:43.970772    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93017d-7d89dbc9d6-tz7zv"]
Sep 17 01:00:44.354910 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:44.354886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:44.591723 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:44.589709    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="310eff3a-f4ac-472e-bc99-1833c4e885a0" path="/var/lib/kubelet/pods/310eff3a-f4ac-472e-bc99-1833c4e885a0/volumes"
Sep 17 01:00:44.591723 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:44.590297    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33082d64-2f02-408e-b355-0d63dac4f89d" path="/var/lib/kubelet/pods/33082d64-2f02-408e-b355-0d63dac4f89d/volumes"
Sep 17 01:00:44.591723 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:44.590710    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb44e21f-cae7-4edf-8d27-b4398c8ee8ed" path="/var/lib/kubelet/pods/bb44e21f-cae7-4edf-8d27-b4398c8ee8ed/volumes"
Sep 17 01:00:44.591723 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:44.591108    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edd2d31b-d8b6-49ed-b25e-6867b8e9c513" path="/var/lib/kubelet/pods/edd2d31b-d8b6-49ed-b25e-6867b8e9c513/volumes"
Sep 17 01:00:44.979613 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:44.979585    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:49.900042 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:49.900001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:49.966504 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:49.966471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:00:50.022618 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:00:50.022559    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:01:04.927085 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:04.927053    2576 scope.go:117] "RemoveContainer" containerID="202c4e978c94ddc6096bb34668fe69d356f4b0a1fd53d9291715bc3a06120020"
Sep 17 01:01:04.935682 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:04.935652    2576 scope.go:117] "RemoveContainer" containerID="c6c216afdab0e84ba2bcbdb3966d055677b3a80a01e10cab1420a2adaa52443e"
Sep 17 01:01:10.814101 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:10.814069    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"]
Sep 17 01:01:10.833832 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:10.833808    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:01:11.698423 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:11.698390    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"]
Sep 17 01:01:13.056651 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:13.056621    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:01:25.834210 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:25.834111    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt" podUID="8946bed9-87a6-41fa-95cf-59a2e1e44621" containerName="main" containerID="cri-o://e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844" gracePeriod=45
Sep 17 01:01:26.084970 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.084915    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 01:01:26.165416 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.165387    2576 generic.go:356] "Generic (PLEG): container finished" podID="8946bed9-87a6-41fa-95cf-59a2e1e44621" containerID="e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844" exitCode=0
Sep 17 01:01:26.165559 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.165427    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"
Sep 17 01:01:26.165559 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.165440    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt" event={"ID":"8946bed9-87a6-41fa-95cf-59a2e1e44621","Type":"ContainerDied","Data":"e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844"}
Sep 17 01:01:26.165559 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.165466    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt" event={"ID":"8946bed9-87a6-41fa-95cf-59a2e1e44621","Type":"ContainerDied","Data":"59175b7730de86dfb168117ff5f8ddc591e974330450813ea86893427a1d9744"}
Sep 17 01:01:26.165559 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.165481    2576 scope.go:117] "RemoveContainer" containerID="e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844"
Sep 17 01:01:26.174323 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.174284    2576 scope.go:117] "RemoveContainer" containerID="e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844"
Sep 17 01:01:26.174588 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:01:26.174560    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844\": container with ID starting with e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844 not found: ID does not exist" containerID="e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844"
Sep 17 01:01:26.174659 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.174599    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844"} err="failed to get container status \"e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844\": rpc error: code = NotFound desc = could not find container \"e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844\": container with ID starting with e36c611cb16a539027cc6a10a9dadcd73a215447e888a32f0405382884293844 not found: ID does not exist"
Sep 17 01:01:26.204895 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.204877    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-dshm\") pod \"8946bed9-87a6-41fa-95cf-59a2e1e44621\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") "
Sep 17 01:01:26.204993 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.204902    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-home\") pod \"8946bed9-87a6-41fa-95cf-59a2e1e44621\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") "
Sep 17 01:01:26.204993 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.204973    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8946bed9-87a6-41fa-95cf-59a2e1e44621-tls-certs\") pod \"8946bed9-87a6-41fa-95cf-59a2e1e44621\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") "
Sep 17 01:01:26.205103 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.205009    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d62hc\" (UniqueName: \"kubernetes.io/projected/8946bed9-87a6-41fa-95cf-59a2e1e44621-kube-api-access-d62hc\") pod \"8946bed9-87a6-41fa-95cf-59a2e1e44621\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") "
Sep 17 01:01:26.205103 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.205034    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-model-cache\") pod \"8946bed9-87a6-41fa-95cf-59a2e1e44621\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") "
Sep 17 01:01:26.205103 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.205067    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-tmp-dir\") pod \"8946bed9-87a6-41fa-95cf-59a2e1e44621\" (UID: \"8946bed9-87a6-41fa-95cf-59a2e1e44621\") "
Sep 17 01:01:26.205313 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.205265    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-home" (OuterVolumeSpecName: "home") pod "8946bed9-87a6-41fa-95cf-59a2e1e44621" (UID: "8946bed9-87a6-41fa-95cf-59a2e1e44621"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:01:26.205463 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.205436    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8946bed9-87a6-41fa-95cf-59a2e1e44621" (UID: "8946bed9-87a6-41fa-95cf-59a2e1e44621"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:01:26.205463 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.205446    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:26.205809 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.205510    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-model-cache" (OuterVolumeSpecName: "model-cache") pod "8946bed9-87a6-41fa-95cf-59a2e1e44621" (UID: "8946bed9-87a6-41fa-95cf-59a2e1e44621"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:01:26.207277 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.207255    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8946bed9-87a6-41fa-95cf-59a2e1e44621-kube-api-access-d62hc" (OuterVolumeSpecName: "kube-api-access-d62hc") pod "8946bed9-87a6-41fa-95cf-59a2e1e44621" (UID: "8946bed9-87a6-41fa-95cf-59a2e1e44621"). InnerVolumeSpecName "kube-api-access-d62hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:01:26.207360 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.207282    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-dshm" (OuterVolumeSpecName: "dshm") pod "8946bed9-87a6-41fa-95cf-59a2e1e44621" (UID: "8946bed9-87a6-41fa-95cf-59a2e1e44621"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:01:26.207360 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.207292    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8946bed9-87a6-41fa-95cf-59a2e1e44621-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8946bed9-87a6-41fa-95cf-59a2e1e44621" (UID: "8946bed9-87a6-41fa-95cf-59a2e1e44621"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:01:26.306100 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.306077    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d62hc\" (UniqueName: \"kubernetes.io/projected/8946bed9-87a6-41fa-95cf-59a2e1e44621-kube-api-access-d62hc\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:26.306100 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.306098    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:26.306234 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.306107    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:26.306234 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.306115    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8946bed9-87a6-41fa-95cf-59a2e1e44621-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:26.306234 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.306123    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8946bed9-87a6-41fa-95cf-59a2e1e44621-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:26.489420 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.489388    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"]
Sep 17 01:01:26.491179 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.491159    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/sub-test-model-d2326d-kserve-7b6fbbbcf7-s4jjt"]
Sep 17 01:01:26.587553 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.587528    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8946bed9-87a6-41fa-95cf-59a2e1e44621" path="/var/lib/kubelet/pods/8946bed9-87a6-41fa-95cf-59a2e1e44621/volumes"
Sep 17 01:01:26.718270 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.718243    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949" podUID="5cfc553c-210d-4c52-8839-c1650284dbd5" containerName="main" containerID="cri-o://f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89" gracePeriod=45
Sep 17 01:01:26.970252 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:26.970231    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:01:27.113190 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113108    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfc553c-210d-4c52-8839-c1650284dbd5-tls-certs\") pod \"5cfc553c-210d-4c52-8839-c1650284dbd5\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") "
Sep 17 01:01:27.113190 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113164    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-tmp-dir\") pod \"5cfc553c-210d-4c52-8839-c1650284dbd5\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") "
Sep 17 01:01:27.113190 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113187    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g94rw\" (UniqueName: \"kubernetes.io/projected/5cfc553c-210d-4c52-8839-c1650284dbd5-kube-api-access-g94rw\") pod \"5cfc553c-210d-4c52-8839-c1650284dbd5\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") "
Sep 17 01:01:27.113438 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113207    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-dshm\") pod \"5cfc553c-210d-4c52-8839-c1650284dbd5\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") "
Sep 17 01:01:27.113438 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113331    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-model-cache\") pod \"5cfc553c-210d-4c52-8839-c1650284dbd5\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") "
Sep 17 01:01:27.113438 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113377    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-home\") pod \"5cfc553c-210d-4c52-8839-c1650284dbd5\" (UID: \"5cfc553c-210d-4c52-8839-c1650284dbd5\") "
Sep 17 01:01:27.113438 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113382    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cfc553c-210d-4c52-8839-c1650284dbd5" (UID: "5cfc553c-210d-4c52-8839-c1650284dbd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:01:27.113658 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113574    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cfc553c-210d-4c52-8839-c1650284dbd5" (UID: "5cfc553c-210d-4c52-8839-c1650284dbd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:01:27.113658 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113605    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-home" (OuterVolumeSpecName: "home") pod "5cfc553c-210d-4c52-8839-c1650284dbd5" (UID: "5cfc553c-210d-4c52-8839-c1650284dbd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:01:27.113860 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113769    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:27.114050 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113896    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:27.114050 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.113969    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:27.115411 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.115387    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-dshm" (OuterVolumeSpecName: "dshm") pod "5cfc553c-210d-4c52-8839-c1650284dbd5" (UID: "5cfc553c-210d-4c52-8839-c1650284dbd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:01:27.115517 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.115466    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfc553c-210d-4c52-8839-c1650284dbd5-kube-api-access-g94rw" (OuterVolumeSpecName: "kube-api-access-g94rw") pod "5cfc553c-210d-4c52-8839-c1650284dbd5" (UID: "5cfc553c-210d-4c52-8839-c1650284dbd5"). InnerVolumeSpecName "kube-api-access-g94rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:01:27.115517 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.115473    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cfc553c-210d-4c52-8839-c1650284dbd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cfc553c-210d-4c52-8839-c1650284dbd5" (UID: "5cfc553c-210d-4c52-8839-c1650284dbd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:01:27.171510 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.171483    2576 generic.go:356] "Generic (PLEG): container finished" podID="5cfc553c-210d-4c52-8839-c1650284dbd5" containerID="f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89" exitCode=0
Sep 17 01:01:27.171623 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.171524    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"
Sep 17 01:01:27.171623 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.171581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949" event={"ID":"5cfc553c-210d-4c52-8839-c1650284dbd5","Type":"ContainerDied","Data":"f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89"}
Sep 17 01:01:27.171623 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.171620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949" event={"ID":"5cfc553c-210d-4c52-8839-c1650284dbd5","Type":"ContainerDied","Data":"a73323e93592244559c28ae16a52591bfeedcfeddb2a827987987a45581ed0e4"}
Sep 17 01:01:27.171782 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.171637    2576 scope.go:117] "RemoveContainer" containerID="f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89"
Sep 17 01:01:27.180542 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.180528    2576 scope.go:117] "RemoveContainer" containerID="f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89"
Sep 17 01:01:27.180778 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:01:27.180761    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89\": container with ID starting with f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89 not found: ID does not exist" containerID="f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89"
Sep 17 01:01:27.180837 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.180785    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89"} err="failed to get container status \"f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89\": rpc error: code = NotFound desc = could not find container \"f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89\": container with ID starting with f92c181786193e38438d251330b0c065d76826cfaa760956be7fa7cff8483c89 not found: ID does not exist"
Sep 17 01:01:27.191530 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.191509    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"]
Sep 17 01:01:27.194895 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.194875    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/sub-test-model-bcd112-kserve-59f644f597-hn949"]
Sep 17 01:01:27.215374 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.215354    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfc553c-210d-4c52-8839-c1650284dbd5-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:27.215436 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.215377    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g94rw\" (UniqueName: \"kubernetes.io/projected/5cfc553c-210d-4c52-8839-c1650284dbd5-kube-api-access-g94rw\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:27.215436 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.215386    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfc553c-210d-4c52-8839-c1650284dbd5-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:01:27.293938 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.293912    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"]
Sep 17 01:01:27.294356 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294338    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33082d64-2f02-408e-b355-0d63dac4f89d" containerName="payload-pre-processing"
Sep 17 01:01:27.294450 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294357    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="33082d64-2f02-408e-b355-0d63dac4f89d" containerName="payload-pre-processing"
Sep 17 01:01:27.294450 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294387    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd2d31b-d8b6-49ed-b25e-6867b8e9c513" containerName="revoke-keys"
Sep 17 01:01:27.294450 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294396    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd2d31b-d8b6-49ed-b25e-6867b8e9c513" containerName="revoke-keys"
Sep 17 01:01:27.294450 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294409    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310eff3a-f4ac-472e-bc99-1833c4e885a0" containerName="payload-processing"
Sep 17 01:01:27.294450 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294418    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="310eff3a-f4ac-472e-bc99-1833c4e885a0" containerName="payload-processing"
Sep 17 01:01:27.294450 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294429    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cfc553c-210d-4c52-8839-c1650284dbd5" containerName="main"
Sep 17 01:01:27.294450 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294437    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cfc553c-210d-4c52-8839-c1650284dbd5" containerName="main"
Sep 17 01:01:27.294797 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294460    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8946bed9-87a6-41fa-95cf-59a2e1e44621" containerName="main"
Sep 17 01:01:27.294797 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294468    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8946bed9-87a6-41fa-95cf-59a2e1e44621" containerName="main"
Sep 17 01:01:27.294797 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294556    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd2d31b-d8b6-49ed-b25e-6867b8e9c513" containerName="revoke-keys"
Sep 17 01:01:27.294797 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294570    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8946bed9-87a6-41fa-95cf-59a2e1e44621" containerName="main"
Sep 17 01:01:27.294797 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294590    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cfc553c-210d-4c52-8839-c1650284dbd5" containerName="main"
Sep 17 01:01:27.294797 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294599    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="33082d64-2f02-408e-b355-0d63dac4f89d" containerName="payload-pre-processing"
Sep 17 01:01:27.294797 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.294611    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="310eff3a-f4ac-472e-bc99-1833c4e885a0" containerName="payload-processing"
Sep 17 01:01:27.299411 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.299390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.301885 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.301854    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"kube-root-ca.crt\""
Sep 17 01:01:27.302018 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.301910    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"default-dockercfg-qjg2n\""
Sep 17 01:01:27.302018 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.301963    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"openshift-service-ca.crt\""
Sep 17 01:01:27.302018 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.301864    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"rate-test-model-d2326d-kserve-self-signed-certs\""
Sep 17 01:01:27.305688 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.305668    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"]
Sep 17 01:01:27.316721 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.316695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-tmp-dir\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.316841 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.316738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-home\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.316841 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.316762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-model-cache\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.316841 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.316790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-dshm\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.316841 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.316824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1554fe9f-bb4b-479b-87bb-709592dc561f-tls-certs\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.317060 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.316861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvc8\" (UniqueName: \"kubernetes.io/projected/1554fe9f-bb4b-479b-87bb-709592dc561f-kube-api-access-mvvc8\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.318500 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.318480    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:01:27.417444 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.417370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-tmp-dir\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.417444 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.417405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-home\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.417444 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.417426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-model-cache\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.417677 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.417452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-dshm\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.417677 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.417485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1554fe9f-bb4b-479b-87bb-709592dc561f-tls-certs\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.417677 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.417519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvc8\" (UniqueName: \"kubernetes.io/projected/1554fe9f-bb4b-479b-87bb-709592dc561f-kube-api-access-mvvc8\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.417837 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.417772    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-tmp-dir\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.417837 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.417801    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-model-cache\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.417946 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.417847    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-home\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.419865 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.419845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-dshm\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.420168 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.420132    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1554fe9f-bb4b-479b-87bb-709592dc561f-tls-certs\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.425812 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.425795    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvc8\" (UniqueName: \"kubernetes.io/projected/1554fe9f-bb4b-479b-87bb-709592dc561f-kube-api-access-mvvc8\") pod \"rate-test-model-d2326d-kserve-557b6575cf-rhtzc\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.612360 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.612332    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:27.888313 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:27.888283    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"]
Sep 17 01:01:27.890702 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:01:27.890668    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1554fe9f_bb4b_479b_87bb_709592dc561f.slice/crio-73d6aa9e8540c4b522d0dafbc07237c4efc19de119cf963a0fa312ed9f54da76 WatchSource:0}: Error finding container 73d6aa9e8540c4b522d0dafbc07237c4efc19de119cf963a0fa312ed9f54da76: Status 404 returned error can't find the container with id 73d6aa9e8540c4b522d0dafbc07237c4efc19de119cf963a0fa312ed9f54da76
Sep 17 01:01:28.180753 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:28.180668    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc" event={"ID":"1554fe9f-bb4b-479b-87bb-709592dc561f","Type":"ContainerStarted","Data":"4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b"}
Sep 17 01:01:28.180753 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:28.180706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc" event={"ID":"1554fe9f-bb4b-479b-87bb-709592dc561f","Type":"ContainerStarted","Data":"73d6aa9e8540c4b522d0dafbc07237c4efc19de119cf963a0fa312ed9f54da76"}
Sep 17 01:01:28.200432 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:28.200377    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc" podStartSLOduration=1.2003622840000001 podStartE2EDuration="1.200362284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:01:28.19777745 +0000 UTC m=+1884.153373792" watchObservedRunningTime="2026-09-17 01:01:28.200362284 +0000 UTC m=+1884.155958647"
Sep 17 01:01:28.587487 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:28.587458    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cfc553c-210d-4c52-8839-c1650284dbd5" path="/var/lib/kubelet/pods/5cfc553c-210d-4c52-8839-c1650284dbd5/volumes"
Sep 17 01:01:37.612478 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:37.612446    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:37.613049 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:37.612493    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:37.624599 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:37.624571    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:38.249023 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:38.248992    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:01:50.799049 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.799009    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"]
Sep 17 01:01:50.803133 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.803109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:50.805598 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.805577    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"openshift-service-ca.crt\""
Sep 17 01:01:50.806644 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.806619    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"kube-root-ca.crt\""
Sep 17 01:01:50.806762 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.806668    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"rate-test-model-bcd112-kserve-self-signed-certs\""
Sep 17 01:01:50.806854 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.806683    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcd112\"/\"default-dockercfg-qxzgf\""
Sep 17 01:01:50.810340 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.810285    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"]
Sep 17 01:01:50.827333 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.827306    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:01:50.903922 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.903899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7m4q\" (UniqueName: \"kubernetes.io/projected/70c18022-9b99-4d78-88f8-1fb8b860529e-kube-api-access-g7m4q\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:50.904049 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.903930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-model-cache\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:50.904049 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.903960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-home\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:50.904049 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.904013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-tmp-dir\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:50.904245 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.904068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c18022-9b99-4d78-88f8-1fb8b860529e-tls-certs\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:50.904245 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:50.904099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-dshm\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.005508 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.005482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c18022-9b99-4d78-88f8-1fb8b860529e-tls-certs\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.005658 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.005515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-dshm\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.005658 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.005569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7m4q\" (UniqueName: \"kubernetes.io/projected/70c18022-9b99-4d78-88f8-1fb8b860529e-kube-api-access-g7m4q\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.005658 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.005585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-model-cache\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.005658 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.005616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-home\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.005818 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.005745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-tmp-dir\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.006051 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.006018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-home\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.006184 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.006021    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-model-cache\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.006184 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.006172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-tmp-dir\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.008003 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.007983    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-dshm\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.008199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.008184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c18022-9b99-4d78-88f8-1fb8b860529e-tls-certs\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.013104 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.013079    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7m4q\" (UniqueName: \"kubernetes.io/projected/70c18022-9b99-4d78-88f8-1fb8b860529e-kube-api-access-g7m4q\") pod \"rate-test-model-bcd112-kserve-9647b999f-mmb45\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") " pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.114327 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.114267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:01:51.386338 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:51.386315    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"]
Sep 17 01:01:51.388219 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:01:51.388176    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70c18022_9b99_4d78_88f8_1fb8b860529e.slice/crio-b162b382fbb00ab74d1ff386e949c2ecea3a422e1916afed3d98f4a5e4ac1d8f WatchSource:0}: Error finding container b162b382fbb00ab74d1ff386e949c2ecea3a422e1916afed3d98f4a5e4ac1d8f: Status 404 returned error can't find the container with id b162b382fbb00ab74d1ff386e949c2ecea3a422e1916afed3d98f4a5e4ac1d8f
Sep 17 01:01:52.310779 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:52.310737    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45" event={"ID":"70c18022-9b99-4d78-88f8-1fb8b860529e","Type":"ContainerStarted","Data":"3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48"}
Sep 17 01:01:52.311169 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:52.310791    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45" event={"ID":"70c18022-9b99-4d78-88f8-1fb8b860529e","Type":"ContainerStarted","Data":"b162b382fbb00ab74d1ff386e949c2ecea3a422e1916afed3d98f4a5e4ac1d8f"}
Sep 17 01:01:52.328345 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:01:52.328293    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45" podStartSLOduration=2.328277394 podStartE2EDuration="2.328277394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:01:52.327765093 +0000 UTC m=+1908.283361435" watchObservedRunningTime="2026-09-17 01:01:52.328277394 +0000 UTC m=+1908.283873735"
Sep 17 01:02:01.114872 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:01.114836    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:02:01.115426 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:01.114893    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:02:01.127110 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:01.127088    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:02:01.379290 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:01.379214    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:02:23.828985 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:23.828943    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:43.379854 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:43.379816    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"]
Sep 17 01:02:43.397264 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:43.397118    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:44.376774 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:44.376739    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"]
Sep 17 01:02:44.395246 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:44.395220    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:44.836456 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:44.836422    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:45.115360 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:45.115250    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:45.521764 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:45.521646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:47.055055 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:47.055009    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:47.761305 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:47.761270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:47.877651 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:47.877620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:47.982998 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:47.982972    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:48.568382 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:48.568354    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:49.365573 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:49.365523    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:49.462079 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:49.462028    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:49.608028 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:49.608000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:49.906424 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:49.906395    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:50.007509 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.007481    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:02:50.040607 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.040576    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"]
Sep 17 01:02:50.044230 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.044209    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:02:50.046740 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.046721    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e9533\""
Sep 17 01:02:50.046869 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.046763    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7e9533-dockercfg-5rr7k\""
Sep 17 01:02:50.053778 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.053757    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"]
Sep 17 01:02:50.082590 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.082557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8ad301-7822-4c29-8895-72a1643d0d7d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx\" (UID: \"5b8ad301-7822-4c29-8895-72a1643d0d7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:02:50.082720 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.082608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjdtb\" (UniqueName: \"kubernetes.io/projected/5b8ad301-7822-4c29-8895-72a1643d0d7d-kube-api-access-pjdtb\") pod \"payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx\" (UID: \"5b8ad301-7822-4c29-8895-72a1643d0d7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:02:50.183460 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.183378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8ad301-7822-4c29-8895-72a1643d0d7d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx\" (UID: \"5b8ad301-7822-4c29-8895-72a1643d0d7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:02:50.183460 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.183420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjdtb\" (UniqueName: \"kubernetes.io/projected/5b8ad301-7822-4c29-8895-72a1643d0d7d-kube-api-access-pjdtb\") pod \"payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx\" (UID: \"5b8ad301-7822-4c29-8895-72a1643d0d7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:02:50.187165 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.184584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8ad301-7822-4c29-8895-72a1643d0d7d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx\" (UID: \"5b8ad301-7822-4c29-8895-72a1643d0d7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:02:50.193173 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.193109    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjdtb\" (UniqueName: \"kubernetes.io/projected/5b8ad301-7822-4c29-8895-72a1643d0d7d-kube-api-access-pjdtb\") pod \"payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx\" (UID: \"5b8ad301-7822-4c29-8895-72a1643d0d7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:02:50.355552 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.355522    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:02:50.626300 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.626276    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"]
Sep 17 01:02:50.627440 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:02:50.627415    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8ad301_7822_4c29_8895_72a1643d0d7d.slice/crio-0a183b313cbfba3b715dbb156a91dc183cf25a130080478382fad13bcf428ea5 WatchSource:0}: Error finding container 0a183b313cbfba3b715dbb156a91dc183cf25a130080478382fad13bcf428ea5: Status 404 returned error can't find the container with id 0a183b313cbfba3b715dbb156a91dc183cf25a130080478382fad13bcf428ea5
Sep 17 01:02:50.634405 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:50.634376    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx" event={"ID":"5b8ad301-7822-4c29-8895-72a1643d0d7d","Type":"ContainerStarted","Data":"0a183b313cbfba3b715dbb156a91dc183cf25a130080478382fad13bcf428ea5"}
Sep 17 01:02:51.642590 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:51.642548    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx" event={"ID":"5b8ad301-7822-4c29-8895-72a1643d0d7d","Type":"ContainerStarted","Data":"88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2"}
Sep 17 01:02:51.643058 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:51.642715    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:02:51.659176 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:51.659115    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx" podStartSLOduration=1.659101567 podStartE2EDuration="1.659101567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:02:51.657390098 +0000 UTC m=+1967.612986440" watchObservedRunningTime="2026-09-17 01:02:51.659101567 +0000 UTC m=+1967.614697942"
Sep 17 01:02:58.402214 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.402121    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc" podUID="1554fe9f-bb4b-479b-87bb-709592dc561f" containerName="main" containerID="cri-o://4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b" gracePeriod=45
Sep 17 01:02:58.666889 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.666829    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:02:58.680645 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.680618    2576 generic.go:356] "Generic (PLEG): container finished" podID="1554fe9f-bb4b-479b-87bb-709592dc561f" containerID="4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b" exitCode=0
Sep 17 01:02:58.680792 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.680659    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"
Sep 17 01:02:58.680792 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.680665    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc" event={"ID":"1554fe9f-bb4b-479b-87bb-709592dc561f","Type":"ContainerDied","Data":"4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b"}
Sep 17 01:02:58.680792 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.680705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc" event={"ID":"1554fe9f-bb4b-479b-87bb-709592dc561f","Type":"ContainerDied","Data":"73d6aa9e8540c4b522d0dafbc07237c4efc19de119cf963a0fa312ed9f54da76"}
Sep 17 01:02:58.680792 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.680729    2576 scope.go:117] "RemoveContainer" containerID="4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b"
Sep 17 01:02:58.691104 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.691086    2576 scope.go:117] "RemoveContainer" containerID="4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b"
Sep 17 01:02:58.691415 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:02:58.691387    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b\": container with ID starting with 4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b not found: ID does not exist" containerID="4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b"
Sep 17 01:02:58.691519 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.691425    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b"} err="failed to get container status \"4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b\": rpc error: code = NotFound desc = could not find container \"4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b\": container with ID starting with 4dbd1a866feb16b81a9572704ec2d4591e0a473838e334c891706cef4ed7a14b not found: ID does not exist"
Sep 17 01:02:58.755849 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.755829    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-home\") pod \"1554fe9f-bb4b-479b-87bb-709592dc561f\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") "
Sep 17 01:02:58.755943 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.755869    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-model-cache\") pod \"1554fe9f-bb4b-479b-87bb-709592dc561f\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") "
Sep 17 01:02:58.755943 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.755895    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-tmp-dir\") pod \"1554fe9f-bb4b-479b-87bb-709592dc561f\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") "
Sep 17 01:02:58.755943 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.755931    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-dshm\") pod \"1554fe9f-bb4b-479b-87bb-709592dc561f\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") "
Sep 17 01:02:58.756084 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.755962    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1554fe9f-bb4b-479b-87bb-709592dc561f-tls-certs\") pod \"1554fe9f-bb4b-479b-87bb-709592dc561f\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") "
Sep 17 01:02:58.756084 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.756025    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvvc8\" (UniqueName: \"kubernetes.io/projected/1554fe9f-bb4b-479b-87bb-709592dc561f-kube-api-access-mvvc8\") pod \"1554fe9f-bb4b-479b-87bb-709592dc561f\" (UID: \"1554fe9f-bb4b-479b-87bb-709592dc561f\") "
Sep 17 01:02:58.756229 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.756112    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-home" (OuterVolumeSpecName: "home") pod "1554fe9f-bb4b-479b-87bb-709592dc561f" (UID: "1554fe9f-bb4b-479b-87bb-709592dc561f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:02:58.756229 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.756129    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-model-cache" (OuterVolumeSpecName: "model-cache") pod "1554fe9f-bb4b-479b-87bb-709592dc561f" (UID: "1554fe9f-bb4b-479b-87bb-709592dc561f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:02:58.756229 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.756191    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1554fe9f-bb4b-479b-87bb-709592dc561f" (UID: "1554fe9f-bb4b-479b-87bb-709592dc561f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:02:58.756405 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.756378    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:58.756405 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.756389    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:58.756405 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.756399    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:58.758053 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.758033    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-dshm" (OuterVolumeSpecName: "dshm") pod "1554fe9f-bb4b-479b-87bb-709592dc561f" (UID: "1554fe9f-bb4b-479b-87bb-709592dc561f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:02:58.758292 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.758272    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1554fe9f-bb4b-479b-87bb-709592dc561f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1554fe9f-bb4b-479b-87bb-709592dc561f" (UID: "1554fe9f-bb4b-479b-87bb-709592dc561f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:02:58.758359 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.758336    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1554fe9f-bb4b-479b-87bb-709592dc561f-kube-api-access-mvvc8" (OuterVolumeSpecName: "kube-api-access-mvvc8") pod "1554fe9f-bb4b-479b-87bb-709592dc561f" (UID: "1554fe9f-bb4b-479b-87bb-709592dc561f"). InnerVolumeSpecName "kube-api-access-mvvc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:02:58.857579 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.857556    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1554fe9f-bb4b-479b-87bb-709592dc561f-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:58.857579 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.857576    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1554fe9f-bb4b-479b-87bb-709592dc561f-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:58.857701 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:58.857587    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvvc8\" (UniqueName: \"kubernetes.io/projected/1554fe9f-bb4b-479b-87bb-709592dc561f-kube-api-access-mvvc8\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:59.002587 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.002553    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"]
Sep 17 01:02:59.005462 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.005440    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/rate-test-model-d2326d-kserve-557b6575cf-rhtzc"]
Sep 17 01:02:59.396771 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.396728    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45" podUID="70c18022-9b99-4d78-88f8-1fb8b860529e" containerName="main" containerID="cri-o://3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48" gracePeriod=45
Sep 17 01:02:59.646244 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.646216    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:02:59.688394 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.688316    2576 generic.go:356] "Generic (PLEG): container finished" podID="70c18022-9b99-4d78-88f8-1fb8b860529e" containerID="3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48" exitCode=0
Sep 17 01:02:59.688394 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.688360    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"
Sep 17 01:02:59.688583 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.688406    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45" event={"ID":"70c18022-9b99-4d78-88f8-1fb8b860529e","Type":"ContainerDied","Data":"3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48"}
Sep 17 01:02:59.688583 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.688458    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45" event={"ID":"70c18022-9b99-4d78-88f8-1fb8b860529e","Type":"ContainerDied","Data":"b162b382fbb00ab74d1ff386e949c2ecea3a422e1916afed3d98f4a5e4ac1d8f"}
Sep 17 01:02:59.688583 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.688481    2576 scope.go:117] "RemoveContainer" containerID="3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48"
Sep 17 01:02:59.697212 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.697183    2576 scope.go:117] "RemoveContainer" containerID="3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48"
Sep 17 01:02:59.697456 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:02:59.697438    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48\": container with ID starting with 3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48 not found: ID does not exist" containerID="3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48"
Sep 17 01:02:59.697529 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.697480    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48"} err="failed to get container status \"3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48\": rpc error: code = NotFound desc = could not find container \"3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48\": container with ID starting with 3536f5e5d798d309bb29b14cf8119deb92e7e73e46a37933c5073c8c77694d48 not found: ID does not exist"
Sep 17 01:02:59.765374 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765350    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-model-cache\") pod \"70c18022-9b99-4d78-88f8-1fb8b860529e\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") "
Sep 17 01:02:59.765491 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765444    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-home\") pod \"70c18022-9b99-4d78-88f8-1fb8b860529e\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") "
Sep 17 01:02:59.765558 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765511    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-tmp-dir\") pod \"70c18022-9b99-4d78-88f8-1fb8b860529e\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") "
Sep 17 01:02:59.765558 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765529    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7m4q\" (UniqueName: \"kubernetes.io/projected/70c18022-9b99-4d78-88f8-1fb8b860529e-kube-api-access-g7m4q\") pod \"70c18022-9b99-4d78-88f8-1fb8b860529e\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") "
Sep 17 01:02:59.765558 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765551    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-dshm\") pod \"70c18022-9b99-4d78-88f8-1fb8b860529e\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") "
Sep 17 01:02:59.765709 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765571    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c18022-9b99-4d78-88f8-1fb8b860529e-tls-certs\") pod \"70c18022-9b99-4d78-88f8-1fb8b860529e\" (UID: \"70c18022-9b99-4d78-88f8-1fb8b860529e\") "
Sep 17 01:02:59.765709 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765627    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-model-cache" (OuterVolumeSpecName: "model-cache") pod "70c18022-9b99-4d78-88f8-1fb8b860529e" (UID: "70c18022-9b99-4d78-88f8-1fb8b860529e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:02:59.765709 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765661    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-home" (OuterVolumeSpecName: "home") pod "70c18022-9b99-4d78-88f8-1fb8b860529e" (UID: "70c18022-9b99-4d78-88f8-1fb8b860529e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:02:59.765837 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765792    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70c18022-9b99-4d78-88f8-1fb8b860529e" (UID: "70c18022-9b99-4d78-88f8-1fb8b860529e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:02:59.765907 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765893    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:59.765966 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765909    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:59.765966 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.765919    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:59.767793 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.767756    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-dshm" (OuterVolumeSpecName: "dshm") pod "70c18022-9b99-4d78-88f8-1fb8b860529e" (UID: "70c18022-9b99-4d78-88f8-1fb8b860529e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:02:59.767793 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.767770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c18022-9b99-4d78-88f8-1fb8b860529e-kube-api-access-g7m4q" (OuterVolumeSpecName: "kube-api-access-g7m4q") pod "70c18022-9b99-4d78-88f8-1fb8b860529e" (UID: "70c18022-9b99-4d78-88f8-1fb8b860529e"). InnerVolumeSpecName "kube-api-access-g7m4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:02:59.767793 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.767783    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c18022-9b99-4d78-88f8-1fb8b860529e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70c18022-9b99-4d78-88f8-1fb8b860529e" (UID: "70c18022-9b99-4d78-88f8-1fb8b860529e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:02:59.866938 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.866914    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7m4q\" (UniqueName: \"kubernetes.io/projected/70c18022-9b99-4d78-88f8-1fb8b860529e-kube-api-access-g7m4q\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:59.866938 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.866934    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c18022-9b99-4d78-88f8-1fb8b860529e-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:02:59.867068 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:02:59.866945    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c18022-9b99-4d78-88f8-1fb8b860529e-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:03:00.009136 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:00.009111    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"]
Sep 17 01:03:00.014405 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:00.014384    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcd112/rate-test-model-bcd112-kserve-9647b999f-mmb45"]
Sep 17 01:03:00.588669 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:00.588641    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1554fe9f-bb4b-479b-87bb-709592dc561f" path="/var/lib/kubelet/pods/1554fe9f-bb4b-479b-87bb-709592dc561f/volumes"
Sep 17 01:03:00.589007 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:00.588994    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c18022-9b99-4d78-88f8-1fb8b860529e" path="/var/lib/kubelet/pods/70c18022-9b99-4d78-88f8-1fb8b860529e/volumes"
Sep 17 01:03:02.650954 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:02.650916    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:03:04.989773 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:04.989738    2576 scope.go:117] "RemoveContainer" containerID="e1f6488820a265fbb8661b4dec03ed7a6d847cb26921dc5543bf18dd8b98c134"
Sep 17 01:03:11.355432 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:11.355400    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:12.054957 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:12.054929    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:12.665780 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:12.665743    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:12.777590 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:12.777558    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:13.432956 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.432917    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:13.533774 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.533736    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:13.720496 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.720426    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"]
Sep 17 01:03:13.720851 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.720823    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1554fe9f-bb4b-479b-87bb-709592dc561f" containerName="main"
Sep 17 01:03:13.720851 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.720838    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1554fe9f-bb4b-479b-87bb-709592dc561f" containerName="main"
Sep 17 01:03:13.720975 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.720856    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c18022-9b99-4d78-88f8-1fb8b860529e" containerName="main"
Sep 17 01:03:13.720975 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.720862    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c18022-9b99-4d78-88f8-1fb8b860529e" containerName="main"
Sep 17 01:03:13.720975 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.720934    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c18022-9b99-4d78-88f8-1fb8b860529e" containerName="main"
Sep 17 01:03:13.720975 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.720947    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1554fe9f-bb4b-479b-87bb-709592dc561f" containerName="main"
Sep 17 01:03:13.724119 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.724099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.726923 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.726903    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c13796-openshift-default-dockercfg-gn4jl\""
Sep 17 01:03:13.734589 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.734560    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"]
Sep 17 01:03:13.746240 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.746214    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:13.783581 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.783548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-token\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.783689 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.783592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-credential-socket\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.783689 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.783649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-envoy\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.783689 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.783676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-podinfo\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.783913 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.783747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-data\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.783913 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.783804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6219c715-95cd-49c0-93ee-9351cfe223ad-istiod-ca-cert\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.783913 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.783889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8hn7\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-kube-api-access-z8hn7\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.784051 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.783930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-socket\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.784051 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.783969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-certs\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.884436 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8hn7\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-kube-api-access-z8hn7\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.884624 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-socket\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.884624 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-certs\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.884624 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-token\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.884624 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-credential-socket\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.884624 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-envoy\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.884900 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-podinfo\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.884900 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-data\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.884900 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6219c715-95cd-49c0-93ee-9351cfe223ad-istiod-ca-cert\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.885064 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-certs\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.885064 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.884954    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-socket\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.885188 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.885069    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-data\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.885188 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.885157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-credential-socket\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.885397 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.885380    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6219c715-95cd-49c0-93ee-9351cfe223ad-istiod-ca-cert\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.887509 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.887482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-envoy\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.887838 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.887816    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-podinfo\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.894801 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.894768    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-token\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:13.894964 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:13.894940    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8hn7\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-kube-api-access-z8hn7\") pod \"e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") " pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:14.036785 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:14.036759    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:14.234040 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:14.234006    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:14.340874 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:14.340833    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"]
Sep 17 01:03:14.341756 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:03:14.341724    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6219c715_95cd_49c0_93ee_9351cfe223ad.slice/crio-f1746babf548ff9750c81bfaa64e01f5b31856f7702355215b5f381d4c20eea0 WatchSource:0}: Error finding container f1746babf548ff9750c81bfaa64e01f5b31856f7702355215b5f381d4c20eea0: Status 404 returned error can't find the container with id f1746babf548ff9750c81bfaa64e01f5b31856f7702355215b5f381d4c20eea0
Sep 17 01:03:14.344989 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:14.344953    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 01:03:14.345108 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:14.345040    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 01:03:14.345108 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:14.345085    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 01:03:14.776108 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:14.776026    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" event={"ID":"6219c715-95cd-49c0-93ee-9351cfe223ad","Type":"ContainerStarted","Data":"3ef0f616047a95893bd4160c3905a4496615e08a94e631a9ff8a461897b27672"}
Sep 17 01:03:14.776108 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:14.776104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" event={"ID":"6219c715-95cd-49c0-93ee-9351cfe223ad","Type":"ContainerStarted","Data":"f1746babf548ff9750c81bfaa64e01f5b31856f7702355215b5f381d4c20eea0"}
Sep 17 01:03:14.794190 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:14.794126    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" podStartSLOduration=1.794114301 podStartE2EDuration="1.794114301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:03:14.792876621 +0000 UTC m=+1990.748472972" watchObservedRunningTime="2026-09-17 01:03:14.794114301 +0000 UTC m=+1990.749710642"
Sep 17 01:03:15.037632 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:15.037600    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:15.043284 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:15.043253    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:15.781284 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:15.781244    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:15.782289 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:15.782265    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:16.037509 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:16.037418    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:16.685525 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:16.685495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:16.793216 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:16.793186    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:17.551828 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:17.551796    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:18.197319 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:18.197287    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:18.855256 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:18.855227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:18.955916 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:18.955886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:19.060904 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.060877    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:19.314253 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.314229    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:19.403788 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.403761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:19.408993 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.408967    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"]
Sep 17 01:03:19.412646 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.412625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:19.414915 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.414890    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c13796-dockercfg-r4mjd\""
Sep 17 01:03:19.415023 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.414918    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c13796\""
Sep 17 01:03:19.422531 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.422504    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"]
Sep 17 01:03:19.442950 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.442921    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"]
Sep 17 01:03:19.446978 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.446960    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:19.454875 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.454850    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"]
Sep 17 01:03:19.534976 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.534941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2bbc28-1888-4b4e-ba4a-755428aba06b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l\" (UID: \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:19.535177 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.535012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde081c9-29fd-4a8e-935c-03f5918f2fa3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8\" (UID: \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:19.535177 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.535045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmtb\" (UniqueName: \"kubernetes.io/projected/dde081c9-29fd-4a8e-935c-03f5918f2fa3-kube-api-access-fkmtb\") pod \"payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8\" (UID: \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:19.535177 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.535158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmn96\" (UniqueName: \"kubernetes.io/projected/9e2bbc28-1888-4b4e-ba4a-755428aba06b-kube-api-access-kmn96\") pod \"payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l\" (UID: \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:19.635941 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.635875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmn96\" (UniqueName: \"kubernetes.io/projected/9e2bbc28-1888-4b4e-ba4a-755428aba06b-kube-api-access-kmn96\") pod \"payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l\" (UID: \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:19.635941 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.635906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2bbc28-1888-4b4e-ba4a-755428aba06b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l\" (UID: \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:19.636134 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.635944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde081c9-29fd-4a8e-935c-03f5918f2fa3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8\" (UID: \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:19.636134 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.635969    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmtb\" (UniqueName: \"kubernetes.io/projected/dde081c9-29fd-4a8e-935c-03f5918f2fa3-kube-api-access-fkmtb\") pod \"payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8\" (UID: \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:19.636803 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.636782    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde081c9-29fd-4a8e-935c-03f5918f2fa3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8\" (UID: \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:19.636901 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.636863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2bbc28-1888-4b4e-ba4a-755428aba06b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l\" (UID: \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:19.644324 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.644305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmtb\" (UniqueName: \"kubernetes.io/projected/dde081c9-29fd-4a8e-935c-03f5918f2fa3-kube-api-access-fkmtb\") pod \"payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8\" (UID: \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:19.644445 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.644423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmn96\" (UniqueName: \"kubernetes.io/projected/9e2bbc28-1888-4b4e-ba4a-755428aba06b-kube-api-access-kmn96\") pod \"payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l\" (UID: \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:19.724886 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.724865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:19.758196 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:19.758166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:20.001912 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.001890    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"]
Sep 17 01:03:20.003449 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:03:20.003425    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2bbc28_1888_4b4e_ba4a_755428aba06b.slice/crio-43d1a8d24e9bfe190f0679992600f769558c17961fbab582fe7f6ce52b287f74 WatchSource:0}: Error finding container 43d1a8d24e9bfe190f0679992600f769558c17961fbab582fe7f6ce52b287f74: Status 404 returned error can't find the container with id 43d1a8d24e9bfe190f0679992600f769558c17961fbab582fe7f6ce52b287f74
Sep 17 01:03:20.041566 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.041539    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"]
Sep 17 01:03:20.044945 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:03:20.044916    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde081c9_29fd_4a8e_935c_03f5918f2fa3.slice/crio-9b01633e8297a288b367be6fdcd7a4fd77e241cb8ce3609bda104cf944ec5b08 WatchSource:0}: Error finding container 9b01633e8297a288b367be6fdcd7a4fd77e241cb8ce3609bda104cf944ec5b08: Status 404 returned error can't find the container with id 9b01633e8297a288b367be6fdcd7a4fd77e241cb8ce3609bda104cf944ec5b08
Sep 17 01:03:20.811415 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.811367    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8" event={"ID":"dde081c9-29fd-4a8e-935c-03f5918f2fa3","Type":"ContainerStarted","Data":"e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b"}
Sep 17 01:03:20.811865 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.811423    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8" event={"ID":"dde081c9-29fd-4a8e-935c-03f5918f2fa3","Type":"ContainerStarted","Data":"9b01633e8297a288b367be6fdcd7a4fd77e241cb8ce3609bda104cf944ec5b08"}
Sep 17 01:03:20.811865 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.811453    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:20.813086 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.813057    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l" event={"ID":"9e2bbc28-1888-4b4e-ba4a-755428aba06b","Type":"ContainerStarted","Data":"ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2"}
Sep 17 01:03:20.813222 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.813104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l" event={"ID":"9e2bbc28-1888-4b4e-ba4a-755428aba06b","Type":"ContainerStarted","Data":"43d1a8d24e9bfe190f0679992600f769558c17961fbab582fe7f6ce52b287f74"}
Sep 17 01:03:20.813222 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.813182    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:20.828585 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.828539    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8" podStartSLOduration=1.828526643 podStartE2EDuration="1.828526643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:03:20.827541013 +0000 UTC m=+1996.783137353" watchObservedRunningTime="2026-09-17 01:03:20.828526643 +0000 UTC m=+1996.784122984"
Sep 17 01:03:20.841313 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:20.841268    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l" podStartSLOduration=1.841256407 podStartE2EDuration="1.841256407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:03:20.840679897 +0000 UTC m=+1996.796276250" watchObservedRunningTime="2026-09-17 01:03:20.841256407 +0000 UTC m=+1996.796852750"
Sep 17 01:03:31.819698 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:31.819663    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:03:31.820101 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:31.819838    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:03:34.912424 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:34.912391    2576 generic.go:356] "Generic (PLEG): container finished" podID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerID="3ef0f616047a95893bd4160c3905a4496615e08a94e631a9ff8a461897b27672" exitCode=0
Sep 17 01:03:34.912818 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:34.912477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" event={"ID":"6219c715-95cd-49c0-93ee-9351cfe223ad","Type":"ContainerDied","Data":"3ef0f616047a95893bd4160c3905a4496615e08a94e631a9ff8a461897b27672"}
Sep 17 01:03:34.912860 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:34.912817    2576 scope.go:117] "RemoveContainer" containerID="3ef0f616047a95893bd4160c3905a4496615e08a94e631a9ff8a461897b27672"
Sep 17 01:03:34.913896 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:34.913867    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 01:03:34.913976 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:34.913939    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 01:03:34.913976 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:34.913970    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 01:03:35.919656 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:35.919623    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" event={"ID":"6219c715-95cd-49c0-93ee-9351cfe223ad","Type":"ContainerStarted","Data":"b8f6e9b0da340ce0cae806b6b711d306a665bae886eb4e3b4c73855f6cc8c486"}
Sep 17 01:03:36.037242 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:36.037205    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:36.037573 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:36.037540    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Sep 17 01:03:36.037668 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:36.037613    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Sep 17 01:03:37.037943 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:37.037909    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Sep 17 01:03:37.038559 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:37.038529    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Sep 17 01:03:38.037220 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:38.037179    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Sep 17 01:03:38.037441 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:38.037248    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Sep 17 01:03:39.037948 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:39.037880    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Sep 17 01:03:39.038483 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:39.037980    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Sep 17 01:03:40.037801 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:40.037765    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Sep 17 01:03:40.038095 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:40.038065    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Sep 17 01:03:40.691673 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:40.691637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:41.225883 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:41.225841    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:41.226479 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:41.226456    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:41.227428 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:41.227409    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:03:41.436907 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:41.436871    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:41.648985 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:41.648936    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:42.063243 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:42.063213    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:42.697225 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:42.697178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:43.344866 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:43.344831    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:46.760287 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:46.760259    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8_dde081c9-29fd-4a8e-935c-03f5918f2fa3/payload-processing/0.log"
Sep 17 01:03:56.690676 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.690642    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"]
Sep 17 01:03:56.695609 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.695589    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.697927 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.697905    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7e9533\"/\"ipp-route-7e9533-kserve-self-signed-certs\""
Sep 17 01:03:56.698909 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.698882    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7e9533\"/\"openshift-service-ca.crt\""
Sep 17 01:03:56.699332 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.699025    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7e9533\"/\"kube-root-ca.crt\""
Sep 17 01:03:56.699332 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.699283    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7e9533\"/\"default-dockercfg-clhsj\""
Sep 17 01:03:56.702382 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.702362    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"]
Sep 17 01:03:56.758920 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.758894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-home\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.759045 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.758966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-model-cache\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.759045 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.759000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tls-certs\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.759045 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.759024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4d6b\" (UniqueName: \"kubernetes.io/projected/a23b53ba-41f3-40b3-8576-a7a6abb8e865-kube-api-access-c4d6b\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.759189 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.759102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-dshm\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.759233 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.759197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tmp-dir\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.784370 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.784345    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:03:56.860662 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.860633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tmp-dir\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.860783 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.860693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-home\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.860783 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.860724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-model-cache\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.860783 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.860743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tls-certs\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.860783 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.860759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4d6b\" (UniqueName: \"kubernetes.io/projected/a23b53ba-41f3-40b3-8576-a7a6abb8e865-kube-api-access-c4d6b\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.860783 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.860776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-dshm\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.861105 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.861086    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tmp-dir\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.861194 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.861106    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-model-cache\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.861244 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.861196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-home\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.863070 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.863046    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-dshm\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.863350 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.863333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tls-certs\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:56.868329 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:56.868310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4d6b\" (UniqueName: \"kubernetes.io/projected/a23b53ba-41f3-40b3-8576-a7a6abb8e865-kube-api-access-c4d6b\") pod \"ipp-route-7e9533-kserve-5b78f7fdc5-k8v65\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") " pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:57.006872 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:57.006785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:03:57.290185 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:57.290133    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"]
Sep 17 01:03:57.292576 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:03:57.292549    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda23b53ba_41f3_40b3_8576_a7a6abb8e865.slice/crio-7561f7604c788defbe92437b06576ca79fc914908eee0a07c5c78e39883af7b5 WatchSource:0}: Error finding container 7561f7604c788defbe92437b06576ca79fc914908eee0a07c5c78e39883af7b5: Status 404 returned error can't find the container with id 7561f7604c788defbe92437b06576ca79fc914908eee0a07c5c78e39883af7b5
Sep 17 01:03:58.054416 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:58.054379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65" event={"ID":"a23b53ba-41f3-40b3-8576-a7a6abb8e865","Type":"ContainerStarted","Data":"ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d"}
Sep 17 01:03:58.054815 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:58.054431    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65" event={"ID":"a23b53ba-41f3-40b3-8576-a7a6abb8e865","Type":"ContainerStarted","Data":"7561f7604c788defbe92437b06576ca79fc914908eee0a07c5c78e39883af7b5"}
Sep 17 01:03:58.072091 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:03:58.072041    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65" podStartSLOduration=2.072026357 podStartE2EDuration="2.072026357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:03:58.070305318 +0000 UTC m=+2034.025901660" watchObservedRunningTime="2026-09-17 01:03:58.072026357 +0000 UTC m=+2034.027622697"
Sep 17 01:04:05.019813 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:05.019781    2576 scope.go:117] "RemoveContainer" containerID="22cd6cc0eeba7c33bf11ca58c238876aa0a2b81a2081e87295a868b3e123c156"
Sep 17 01:04:07.006979 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:07.006948    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:04:07.007390 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:07.007041    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:04:07.020046 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:07.020027    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:04:07.119422 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:07.119397    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:04:19.518927 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:19.518894    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx_5b8ad301-7822-4c29-8895-72a1643d0d7d/payload-processing/0.log"
Sep 17 01:04:19.765606 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:19.765578    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8_dde081c9-29fd-4a8e-935c-03f5918f2fa3/payload-processing/0.log"
Sep 17 01:04:20.037107 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:20.037066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:20.337299 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:20.337199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:20.670621 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:20.670533    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:22.555219 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:22.555188    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:23.457827 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:23.457795    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:23.560665 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:23.560594    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:24.449602 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:24.449566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:25.239606 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:25.239573    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:25.863532 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:25.863441    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:26.529692 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:26.529665    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:26.628845 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:26.628813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:26.963050 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:26.962974    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:27.067561 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.067532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:27.107654 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.107625    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"]
Sep 17 01:04:27.111571 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.111553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:27.114100 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.114081    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-9187f3\""
Sep 17 01:04:27.114100 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.114093    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-9187f3-dockercfg-zdfps\""
Sep 17 01:04:27.119800 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.119415    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"]
Sep 17 01:04:27.135318 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.135295    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:27.216997 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.216917    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml\" (UID: \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:27.216997 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.216953    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxvm\" (UniqueName: \"kubernetes.io/projected/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-kube-api-access-lbxvm\") pod \"payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml\" (UID: \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:27.317901 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.317866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml\" (UID: \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:27.317901 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.317903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxvm\" (UniqueName: \"kubernetes.io/projected/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-kube-api-access-lbxvm\") pod \"payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml\" (UID: \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:27.318684 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.318665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml\" (UID: \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:27.325879 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.325855    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxvm\" (UniqueName: \"kubernetes.io/projected/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-kube-api-access-lbxvm\") pod \"payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml\" (UID: \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:27.423006 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.422979    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:27.697332 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:27.697307    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"]
Sep 17 01:04:27.698664 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:04:27.698638    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7e4e6c_da13_4eb6_8975_cb86ac48f391.slice/crio-a1310447141a633e01f2fcb5b2719445a459e5962f2659b1d833298af5b884ff WatchSource:0}: Error finding container a1310447141a633e01f2fcb5b2719445a459e5962f2659b1d833298af5b884ff: Status 404 returned error can't find the container with id a1310447141a633e01f2fcb5b2719445a459e5962f2659b1d833298af5b884ff
Sep 17 01:04:28.223319 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:28.223285    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml" event={"ID":"9e7e4e6c-da13-4eb6-8975-cb86ac48f391","Type":"ContainerStarted","Data":"52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b"}
Sep 17 01:04:28.223503 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:28.223338    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml" event={"ID":"9e7e4e6c-da13-4eb6-8975-cb86ac48f391","Type":"ContainerStarted","Data":"a1310447141a633e01f2fcb5b2719445a459e5962f2659b1d833298af5b884ff"}
Sep 17 01:04:28.223575 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:28.223503    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:28.243077 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:28.243023    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml" podStartSLOduration=1.2430047100000001 podStartE2EDuration="1.24300471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:04:28.241367539 +0000 UTC m=+2064.196963878" watchObservedRunningTime="2026-09-17 01:04:28.24300471 +0000 UTC m=+2064.198601052"
Sep 17 01:04:30.712889 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:30.712857    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:39.230739 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:39.230704    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:47.351272 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:47.351234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:48.360213 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:48.360180    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:48.700675 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:48.700591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:49.061681 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.061649    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"]
Sep 17 01:04:49.065227 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.065210    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"
Sep 17 01:04:49.067414 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.067387    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 01:04:49.071744 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.071579    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"]
Sep 17 01:04:49.101502 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.101480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32d7a9d-67b3-41ca-a632-fb22dff5fe10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz\" (UID: \"a32d7a9d-67b3-41ca-a632-fb22dff5fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"
Sep 17 01:04:49.202014 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.201966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32d7a9d-67b3-41ca-a632-fb22dff5fe10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz\" (UID: \"a32d7a9d-67b3-41ca-a632-fb22dff5fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"
Sep 17 01:04:49.202627 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.202604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32d7a9d-67b3-41ca-a632-fb22dff5fe10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz\" (UID: \"a32d7a9d-67b3-41ca-a632-fb22dff5fe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"
Sep 17 01:04:49.379056 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.379002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"
Sep 17 01:04:49.499070 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.499039    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:49.649091 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:49.649066    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"]
Sep 17 01:04:49.651168 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:04:49.651118    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32d7a9d_67b3_41ca_a632_fb22dff5fe10.slice/crio-85f9f3f3fc86fd15ba1f67d592bf6dc880576d9c0f3552d32a46c360d58dd3c0 WatchSource:0}: Error finding container 85f9f3f3fc86fd15ba1f67d592bf6dc880576d9c0f3552d32a46c360d58dd3c0: Status 404 returned error can't find the container with id 85f9f3f3fc86fd15ba1f67d592bf6dc880576d9c0f3552d32a46c360d58dd3c0
Sep 17 01:04:50.356133 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:50.356097    2576 generic.go:356] "Generic (PLEG): container finished" podID="a32d7a9d-67b3-41ca-a632-fb22dff5fe10" containerID="1365192d5346893688582a43890693daeda32d828d64f13d47c36d589b28a2e5" exitCode=0
Sep 17 01:04:50.356284 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:50.356177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz" event={"ID":"a32d7a9d-67b3-41ca-a632-fb22dff5fe10","Type":"ContainerDied","Data":"1365192d5346893688582a43890693daeda32d828d64f13d47c36d589b28a2e5"}
Sep 17 01:04:50.356284 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:50.356276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz" event={"ID":"a32d7a9d-67b3-41ca-a632-fb22dff5fe10","Type":"ContainerStarted","Data":"85f9f3f3fc86fd15ba1f67d592bf6dc880576d9c0f3552d32a46c360d58dd3c0"}
Sep 17 01:04:50.459185 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:50.459124    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:51.158336 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:51.158305    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:51.493910 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:51.493890    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"
Sep 17 01:04:51.518771 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:51.518743    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32d7a9d-67b3-41ca-a632-fb22dff5fe10-maas-api-service-ca\") pod \"a32d7a9d-67b3-41ca-a632-fb22dff5fe10\" (UID: \"a32d7a9d-67b3-41ca-a632-fb22dff5fe10\") "
Sep 17 01:04:51.519073 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:51.519051    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32d7a9d-67b3-41ca-a632-fb22dff5fe10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a32d7a9d-67b3-41ca-a632-fb22dff5fe10" (UID: "a32d7a9d-67b3-41ca-a632-fb22dff5fe10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:04:51.519205 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:51.519170    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32d7a9d-67b3-41ca-a632-fb22dff5fe10-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:04:51.544323 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:51.544303    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:51.859691 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:51.859665    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:52.083703 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:52.083668    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:52.185410 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:52.185336    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:52.369009 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:52.368983    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"
Sep 17 01:04:52.369195 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:52.369006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz" event={"ID":"a32d7a9d-67b3-41ca-a632-fb22dff5fe10","Type":"ContainerDied","Data":"85f9f3f3fc86fd15ba1f67d592bf6dc880576d9c0f3552d32a46c360d58dd3c0"}
Sep 17 01:04:52.369195 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:52.369056    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f9f3f3fc86fd15ba1f67d592bf6dc880576d9c0f3552d32a46c360d58dd3c0"
Sep 17 01:04:52.724972 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:52.724941    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:53.370510 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:53.370480    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:54.845344 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:54.845310    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:55.401508 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.401472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:55.444282 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.444247    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"]
Sep 17 01:04:55.444590 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.444557    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml" podUID="9e7e4e6c-da13-4eb6-8975-cb86ac48f391" containerName="payload-processing" containerID="cri-o://52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b" gracePeriod=30
Sep 17 01:04:55.469465 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.469421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:55.527022 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.526997    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:55.594637 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.594609    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"]
Sep 17 01:04:55.596305 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.596280    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:55.596900 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.596883    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9187f3-x9dmz"]
Sep 17 01:04:55.713749 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.713723    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:55.756699 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.756657    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbxvm\" (UniqueName: \"kubernetes.io/projected/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-kube-api-access-lbxvm\") pod \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\" (UID: \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\") "
Sep 17 01:04:55.756870 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.756765    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-plugins-config-volume\") pod \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\" (UID: \"9e7e4e6c-da13-4eb6-8975-cb86ac48f391\") "
Sep 17 01:04:55.757525 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.757492    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e7e4e6c-da13-4eb6-8975-cb86ac48f391" (UID: "9e7e4e6c-da13-4eb6-8975-cb86ac48f391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:04:55.759334 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.759294    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-kube-api-access-lbxvm" (OuterVolumeSpecName: "kube-api-access-lbxvm") pod "9e7e4e6c-da13-4eb6-8975-cb86ac48f391" (UID: "9e7e4e6c-da13-4eb6-8975-cb86ac48f391"). InnerVolumeSpecName "kube-api-access-lbxvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:04:55.760482 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.760456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:55.858474 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.858448    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbxvm\" (UniqueName: \"kubernetes.io/projected/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-kube-api-access-lbxvm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:04:55.858474 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.858470    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7e4e6c-da13-4eb6-8975-cb86ac48f391-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:04:55.989784 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:55.989701    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:56.089791 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.089766    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:56.398504 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.398468    2576 generic.go:356] "Generic (PLEG): container finished" podID="9e7e4e6c-da13-4eb6-8975-cb86ac48f391" containerID="52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b" exitCode=0
Sep 17 01:04:56.398504 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.398502    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"
Sep 17 01:04:56.398714 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.398533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml" event={"ID":"9e7e4e6c-da13-4eb6-8975-cb86ac48f391","Type":"ContainerDied","Data":"52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b"}
Sep 17 01:04:56.398714 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.398587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml" event={"ID":"9e7e4e6c-da13-4eb6-8975-cb86ac48f391","Type":"ContainerDied","Data":"a1310447141a633e01f2fcb5b2719445a459e5962f2659b1d833298af5b884ff"}
Sep 17 01:04:56.398714 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.398612    2576 scope.go:117] "RemoveContainer" containerID="52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b"
Sep 17 01:04:56.408136 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.408116    2576 scope.go:117] "RemoveContainer" containerID="52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b"
Sep 17 01:04:56.408422 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:04:56.408404    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b\": container with ID starting with 52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b not found: ID does not exist" containerID="52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b"
Sep 17 01:04:56.408512 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.408431    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b"} err="failed to get container status \"52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b\": rpc error: code = NotFound desc = could not find container \"52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b\": container with ID starting with 52f61c4ee666baf8b371dbc8c5bd606237266f66683677c36e3b86cedefb613b not found: ID does not exist"
Sep 17 01:04:56.418997 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.418975    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"]
Sep 17 01:04:56.425205 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.425185    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-9187f3-79ccc599c9-jwjml"]
Sep 17 01:04:56.588748 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.588722    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e7e4e6c-da13-4eb6-8975-cb86ac48f391" path="/var/lib/kubelet/pods/9e7e4e6c-da13-4eb6-8975-cb86ac48f391/volumes"
Sep 17 01:04:56.589071 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.589057    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a32d7a9d-67b3-41ca-a632-fb22dff5fe10" path="/var/lib/kubelet/pods/a32d7a9d-67b3-41ca-a632-fb22dff5fe10/volumes"
Sep 17 01:04:56.667301 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:56.667244    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:57.481623 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:57.481572    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:58.249085 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:58.249051    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:04:58.959680 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:04:58.959640    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:01.760705 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:01.760667    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:01.860500 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:01.860466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:01.939562 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:01.939521    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:05.131485 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.131456    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 01:05:05.135403 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.135385    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 01:05:05.564096 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.564068    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"]
Sep 17 01:05:05.564520 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.564508    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7e4e6c-da13-4eb6-8975-cb86ac48f391" containerName="payload-processing"
Sep 17 01:05:05.564566 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.564522    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7e4e6c-da13-4eb6-8975-cb86ac48f391" containerName="payload-processing"
Sep 17 01:05:05.564566 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.564530    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a32d7a9d-67b3-41ca-a632-fb22dff5fe10" containerName="revoke-keys"
Sep 17 01:05:05.564566 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.564535    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a32d7a9d-67b3-41ca-a632-fb22dff5fe10" containerName="revoke-keys"
Sep 17 01:05:05.564664 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.564592    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a32d7a9d-67b3-41ca-a632-fb22dff5fe10" containerName="revoke-keys"
Sep 17 01:05:05.564664 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.564603    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7e4e6c-da13-4eb6-8975-cb86ac48f391" containerName="payload-processing"
Sep 17 01:05:05.569317 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.569300    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"
Sep 17 01:05:05.572180 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.572126    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 01:05:05.575591 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.575566    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"]
Sep 17 01:05:05.642917 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.642894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78afb930-0b33-408a-9ebd-b2a44e704767-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p\" (UID: \"78afb930-0b33-408a-9ebd-b2a44e704767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"
Sep 17 01:05:05.744495 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.744467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78afb930-0b33-408a-9ebd-b2a44e704767-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p\" (UID: \"78afb930-0b33-408a-9ebd-b2a44e704767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"
Sep 17 01:05:05.745010 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.744992    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78afb930-0b33-408a-9ebd-b2a44e704767-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p\" (UID: \"78afb930-0b33-408a-9ebd-b2a44e704767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"
Sep 17 01:05:05.855838 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.855763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:05.882737 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:05.882718    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"
Sep 17 01:05:06.158729 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:06.158704    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"]
Sep 17 01:05:06.160326 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:05:06.160299    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78afb930_0b33_408a_9ebd_b2a44e704767.slice/crio-2f5a6c03b77747dc48a7e38ef89a62ba674464e5416cf9ba67fd3e1463cd73fa WatchSource:0}: Error finding container 2f5a6c03b77747dc48a7e38ef89a62ba674464e5416cf9ba67fd3e1463cd73fa: Status 404 returned error can't find the container with id 2f5a6c03b77747dc48a7e38ef89a62ba674464e5416cf9ba67fd3e1463cd73fa
Sep 17 01:05:06.461094 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:06.461064    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p" event={"ID":"78afb930-0b33-408a-9ebd-b2a44e704767","Type":"ContainerStarted","Data":"1ed429e19e7457e71e1ecc448c96559d4b680edf55a43f9b3a9eec95c3e60a00"}
Sep 17 01:05:06.461248 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:06.461127    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p" event={"ID":"78afb930-0b33-408a-9ebd-b2a44e704767","Type":"ContainerStarted","Data":"2f5a6c03b77747dc48a7e38ef89a62ba674464e5416cf9ba67fd3e1463cd73fa"}
Sep 17 01:05:06.474984 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:06.474938    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p" podStartSLOduration=1.474923085 podStartE2EDuration="1.474923085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:05:06.473947011 +0000 UTC m=+2102.429543357" watchObservedRunningTime="2026-09-17 01:05:06.474923085 +0000 UTC m=+2102.430519427"
Sep 17 01:05:06.658107 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:06.658073    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:07.268520 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:07.268402    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:07.469226 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:07.469188    2576 generic.go:356] "Generic (PLEG): container finished" podID="78afb930-0b33-408a-9ebd-b2a44e704767" containerID="1ed429e19e7457e71e1ecc448c96559d4b680edf55a43f9b3a9eec95c3e60a00" exitCode=0
Sep 17 01:05:07.469413 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:07.469291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p" event={"ID":"78afb930-0b33-408a-9ebd-b2a44e704767","Type":"ContainerDied","Data":"1ed429e19e7457e71e1ecc448c96559d4b680edf55a43f9b3a9eec95c3e60a00"}
Sep 17 01:05:07.891954 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:07.891927    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:08.620360 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:08.620340    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"
Sep 17 01:05:08.648239 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:08.648215    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:08.671367 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:08.671334    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78afb930-0b33-408a-9ebd-b2a44e704767-maas-api-service-ca\") pod \"78afb930-0b33-408a-9ebd-b2a44e704767\" (UID: \"78afb930-0b33-408a-9ebd-b2a44e704767\") "
Sep 17 01:05:08.671687 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:08.671657    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78afb930-0b33-408a-9ebd-b2a44e704767-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78afb930-0b33-408a-9ebd-b2a44e704767" (UID: "78afb930-0b33-408a-9ebd-b2a44e704767"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:05:08.772438 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:08.772367    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78afb930-0b33-408a-9ebd-b2a44e704767-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:05:09.282897 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:09.282858    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:09.483199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:09.483175    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"
Sep 17 01:05:09.483365 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:09.483176    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p" event={"ID":"78afb930-0b33-408a-9ebd-b2a44e704767","Type":"ContainerDied","Data":"2f5a6c03b77747dc48a7e38ef89a62ba674464e5416cf9ba67fd3e1463cd73fa"}
Sep 17 01:05:09.483365 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:09.483305    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f5a6c03b77747dc48a7e38ef89a62ba674464e5416cf9ba67fd3e1463cd73fa"
Sep 17 01:05:21.136687 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.136651    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:21.156467 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.156436    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"]
Sep 17 01:05:21.156888 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.156692    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx" podUID="5b8ad301-7822-4c29-8895-72a1643d0d7d" containerName="payload-processing" containerID="cri-o://88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2" gracePeriod=30
Sep 17 01:05:21.300170 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.300109    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"]
Sep 17 01:05:21.302159 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.302114    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e9533-dzf8p"]
Sep 17 01:05:21.424203 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.424180    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:05:21.553372 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.553338    2576 generic.go:356] "Generic (PLEG): container finished" podID="5b8ad301-7822-4c29-8895-72a1643d0d7d" containerID="88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2" exitCode=0
Sep 17 01:05:21.553540 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.553385    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"
Sep 17 01:05:21.553540 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.553408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx" event={"ID":"5b8ad301-7822-4c29-8895-72a1643d0d7d","Type":"ContainerDied","Data":"88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2"}
Sep 17 01:05:21.553540 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.553448    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx" event={"ID":"5b8ad301-7822-4c29-8895-72a1643d0d7d","Type":"ContainerDied","Data":"0a183b313cbfba3b715dbb156a91dc183cf25a130080478382fad13bcf428ea5"}
Sep 17 01:05:21.553540 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.553469    2576 scope.go:117] "RemoveContainer" containerID="88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2"
Sep 17 01:05:21.562854 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.562828    2576 scope.go:117] "RemoveContainer" containerID="88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2"
Sep 17 01:05:21.563166 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:05:21.563123    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2\": container with ID starting with 88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2 not found: ID does not exist" containerID="88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2"
Sep 17 01:05:21.563258 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.563217    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2"} err="failed to get container status \"88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2\": rpc error: code = NotFound desc = could not find container \"88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2\": container with ID starting with 88dda799a4ddeff863da1cc14419a9897127b0afe31c576865a99d70f92292c2 not found: ID does not exist"
Sep 17 01:05:21.578857 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.578840    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8ad301-7822-4c29-8895-72a1643d0d7d-plugins-config-volume\") pod \"5b8ad301-7822-4c29-8895-72a1643d0d7d\" (UID: \"5b8ad301-7822-4c29-8895-72a1643d0d7d\") "
Sep 17 01:05:21.578940 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.578923    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjdtb\" (UniqueName: \"kubernetes.io/projected/5b8ad301-7822-4c29-8895-72a1643d0d7d-kube-api-access-pjdtb\") pod \"5b8ad301-7822-4c29-8895-72a1643d0d7d\" (UID: \"5b8ad301-7822-4c29-8895-72a1643d0d7d\") "
Sep 17 01:05:21.579233 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.579209    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8ad301-7822-4c29-8895-72a1643d0d7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b8ad301-7822-4c29-8895-72a1643d0d7d" (UID: "5b8ad301-7822-4c29-8895-72a1643d0d7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:05:21.580994 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.580972    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8ad301-7822-4c29-8895-72a1643d0d7d-kube-api-access-pjdtb" (OuterVolumeSpecName: "kube-api-access-pjdtb") pod "5b8ad301-7822-4c29-8895-72a1643d0d7d" (UID: "5b8ad301-7822-4c29-8895-72a1643d0d7d"). InnerVolumeSpecName "kube-api-access-pjdtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:05:21.679736 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.679683    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8ad301-7822-4c29-8895-72a1643d0d7d-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:05:21.679736 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.679704    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjdtb\" (UniqueName: \"kubernetes.io/projected/5b8ad301-7822-4c29-8895-72a1643d0d7d-kube-api-access-pjdtb\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:05:21.875737 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.875711    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"]
Sep 17 01:05:21.879200 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:21.879178    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e9533-5984857db9-mzzxx"]
Sep 17 01:05:22.590643 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:22.590609    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8ad301-7822-4c29-8895-72a1643d0d7d" path="/var/lib/kubelet/pods/5b8ad301-7822-4c29-8895-72a1643d0d7d/volumes"
Sep 17 01:05:22.591022 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:22.590962    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78afb930-0b33-408a-9ebd-b2a44e704767" path="/var/lib/kubelet/pods/78afb930-0b33-408a-9ebd-b2a44e704767/volumes"
Sep 17 01:05:23.153615 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:23.153583    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:23.756061 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:23.756018    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:26.645337 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:26.645303    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"]
Sep 17 01:05:26.741357 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:26.741311    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:31.589962 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:31.589919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:05:41.664775 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:41.664713    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65" podUID="a23b53ba-41f3-40b3-8576-a7a6abb8e865" containerName="main" containerID="cri-o://ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d" gracePeriod=45
Sep 17 01:05:41.927169 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:41.927094    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:05:42.050347 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050200    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4d6b\" (UniqueName: \"kubernetes.io/projected/a23b53ba-41f3-40b3-8576-a7a6abb8e865-kube-api-access-c4d6b\") pod \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") "
Sep 17 01:05:42.050347 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050242    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tmp-dir\") pod \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") "
Sep 17 01:05:42.050347 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050344    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-home\") pod \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") "
Sep 17 01:05:42.050653 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050374    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-model-cache\") pod \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") "
Sep 17 01:05:42.050653 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050410    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tls-certs\") pod \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") "
Sep 17 01:05:42.050653 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050443    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-dshm\") pod \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\" (UID: \"a23b53ba-41f3-40b3-8576-a7a6abb8e865\") "
Sep 17 01:05:42.050653 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050554    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a23b53ba-41f3-40b3-8576-a7a6abb8e865" (UID: "a23b53ba-41f3-40b3-8576-a7a6abb8e865"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:05:42.050868 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050825    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-model-cache" (OuterVolumeSpecName: "model-cache") pod "a23b53ba-41f3-40b3-8576-a7a6abb8e865" (UID: "a23b53ba-41f3-40b3-8576-a7a6abb8e865"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:05:42.050930 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050909    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:05:42.050988 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.050928    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:05:42.051467 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.051418    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-home" (OuterVolumeSpecName: "home") pod "a23b53ba-41f3-40b3-8576-a7a6abb8e865" (UID: "a23b53ba-41f3-40b3-8576-a7a6abb8e865"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:05:42.053081 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.053054    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23b53ba-41f3-40b3-8576-a7a6abb8e865-kube-api-access-c4d6b" (OuterVolumeSpecName: "kube-api-access-c4d6b") pod "a23b53ba-41f3-40b3-8576-a7a6abb8e865" (UID: "a23b53ba-41f3-40b3-8576-a7a6abb8e865"). InnerVolumeSpecName "kube-api-access-c4d6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:05:42.053253 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.053231    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a23b53ba-41f3-40b3-8576-a7a6abb8e865" (UID: "a23b53ba-41f3-40b3-8576-a7a6abb8e865"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:05:42.053345 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.053291    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-dshm" (OuterVolumeSpecName: "dshm") pod "a23b53ba-41f3-40b3-8576-a7a6abb8e865" (UID: "a23b53ba-41f3-40b3-8576-a7a6abb8e865"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:05:42.151698 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.151676    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a23b53ba-41f3-40b3-8576-a7a6abb8e865-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:05:42.151788 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.151701    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:05:42.151788 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.151711    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4d6b\" (UniqueName: \"kubernetes.io/projected/a23b53ba-41f3-40b3-8576-a7a6abb8e865-kube-api-access-c4d6b\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:05:42.151788 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.151720    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a23b53ba-41f3-40b3-8576-a7a6abb8e865-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:05:42.669104 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.669076    2576 generic.go:356] "Generic (PLEG): container finished" podID="a23b53ba-41f3-40b3-8576-a7a6abb8e865" containerID="ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d" exitCode=0
Sep 17 01:05:42.669492 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.669108    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65" event={"ID":"a23b53ba-41f3-40b3-8576-a7a6abb8e865","Type":"ContainerDied","Data":"ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d"}
Sep 17 01:05:42.669492 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.669172    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65" event={"ID":"a23b53ba-41f3-40b3-8576-a7a6abb8e865","Type":"ContainerDied","Data":"7561f7604c788defbe92437b06576ca79fc914908eee0a07c5c78e39883af7b5"}
Sep 17 01:05:42.669492 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.669192    2576 scope.go:117] "RemoveContainer" containerID="ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d"
Sep 17 01:05:42.669492 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.669121    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"
Sep 17 01:05:42.678308 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.678288    2576 scope.go:117] "RemoveContainer" containerID="ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d"
Sep 17 01:05:42.678582 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:05:42.678552    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d\": container with ID starting with ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d not found: ID does not exist" containerID="ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d"
Sep 17 01:05:42.678677 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.678589    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d"} err="failed to get container status \"ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d\": rpc error: code = NotFound desc = could not find container \"ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d\": container with ID starting with ba15139f94ecf6954e7f82b47627a05ed8a8c5944dafacc50139a55817ad149d not found: ID does not exist"
Sep 17 01:05:42.684619 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.684600    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"]
Sep 17 01:05:42.687912 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:42.687891    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7e9533/ipp-route-7e9533-kserve-5b78f7fdc5-k8v65"]
Sep 17 01:05:44.587900 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:05:44.587869    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23b53ba-41f3-40b3-8576-a7a6abb8e865" path="/var/lib/kubelet/pods/a23b53ba-41f3-40b3-8576-a7a6abb8e865/volumes"
Sep 17 01:06:01.423727 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:01.423624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:03.330333 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:03.330293    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:03.737446 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:03.737319    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:05.055085 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:05.055055    2576 scope.go:117] "RemoveContainer" containerID="1882b9551a5486ae4acb81fd4b9231457ee5683506a4b131126180cb39f6a77f"
Sep 17 01:06:05.063537 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:05.063506    2576 scope.go:117] "RemoveContainer" containerID="47f3001e0ea33e269e94abdde01d627b019b8a6985cc33d9abc1cd44d6466c7b"
Sep 17 01:06:05.071012 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:05.070996    2576 scope.go:117] "RemoveContainer" containerID="4151f0c9f93a5946e5fcfd29dfc797b0524856aa05b46a4acccc0eef99f7d78d"
Sep 17 01:06:05.078200 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:05.078184    2576 scope.go:117] "RemoveContainer" containerID="dd138b061fde240400b153818934758bdc5035b9963dbd162ae67195ab200ea3"
Sep 17 01:06:09.313570 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.313536    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:09.415655 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.415628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:09.590408 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.590332    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:09.958869 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.958798    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"]
Sep 17 01:06:09.959219 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.959205    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8ad301-7822-4c29-8895-72a1643d0d7d" containerName="payload-processing"
Sep 17 01:06:09.959269 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.959220    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8ad301-7822-4c29-8895-72a1643d0d7d" containerName="payload-processing"
Sep 17 01:06:09.959269 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.959229    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23b53ba-41f3-40b3-8576-a7a6abb8e865" containerName="main"
Sep 17 01:06:09.959269 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.959235    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23b53ba-41f3-40b3-8576-a7a6abb8e865" containerName="main"
Sep 17 01:06:09.959269 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.959249    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78afb930-0b33-408a-9ebd-b2a44e704767" containerName="revoke-keys"
Sep 17 01:06:09.959269 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.959255    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78afb930-0b33-408a-9ebd-b2a44e704767" containerName="revoke-keys"
Sep 17 01:06:09.959428 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.959318    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="78afb930-0b33-408a-9ebd-b2a44e704767" containerName="revoke-keys"
Sep 17 01:06:09.959428 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.959328    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23b53ba-41f3-40b3-8576-a7a6abb8e865" containerName="main"
Sep 17 01:06:09.959428 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.959336    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8ad301-7822-4c29-8895-72a1643d0d7d" containerName="payload-processing"
Sep 17 01:06:09.962536 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.962519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"
Sep 17 01:06:09.964738 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.964717    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 01:06:09.968512 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:09.968490    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"]
Sep 17 01:06:10.078906 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:10.078877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48a113-875b-47a7-a45b-1ae64154cf78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6\" (UID: \"cd48a113-875b-47a7-a45b-1ae64154cf78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"
Sep 17 01:06:10.179867 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:10.179841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48a113-875b-47a7-a45b-1ae64154cf78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6\" (UID: \"cd48a113-875b-47a7-a45b-1ae64154cf78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"
Sep 17 01:06:10.180475 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:10.180453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48a113-875b-47a7-a45b-1ae64154cf78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6\" (UID: \"cd48a113-875b-47a7-a45b-1ae64154cf78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"
Sep 17 01:06:10.245254 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:10.245191    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:10.275097 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:10.275072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"
Sep 17 01:06:10.547430 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:10.547405    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"]
Sep 17 01:06:10.549441 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:06:10.549415    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd48a113_875b_47a7_a45b_1ae64154cf78.slice/crio-d45ea3e3e0fb7cc04f0c7806d7c6e793b6a6f7f6a28e14bb6a1e5e895a0a2bec WatchSource:0}: Error finding container d45ea3e3e0fb7cc04f0c7806d7c6e793b6a6f7f6a28e14bb6a1e5e895a0a2bec: Status 404 returned error can't find the container with id d45ea3e3e0fb7cc04f0c7806d7c6e793b6a6f7f6a28e14bb6a1e5e895a0a2bec
Sep 17 01:06:10.822525 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:10.822465    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6" event={"ID":"cd48a113-875b-47a7-a45b-1ae64154cf78","Type":"ContainerStarted","Data":"863661966d87aeb54987f4ecb0f201a8bd322d4194fa2ea698098cda891c3923"}
Sep 17 01:06:10.822525 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:10.822511    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6" event={"ID":"cd48a113-875b-47a7-a45b-1ae64154cf78","Type":"ContainerStarted","Data":"d45ea3e3e0fb7cc04f0c7806d7c6e793b6a6f7f6a28e14bb6a1e5e895a0a2bec"}
Sep 17 01:06:10.837778 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:10.837724    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6" podStartSLOduration=1.837704436 podStartE2EDuration="1.837704436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:06:10.836014875 +0000 UTC m=+2166.791611217" watchObservedRunningTime="2026-09-17 01:06:10.837704436 +0000 UTC m=+2166.793300778"
Sep 17 01:06:11.049205 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:11.049163    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:11.651527 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:11.651493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:11.829217 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:11.829181    2576 generic.go:356] "Generic (PLEG): container finished" podID="cd48a113-875b-47a7-a45b-1ae64154cf78" containerID="863661966d87aeb54987f4ecb0f201a8bd322d4194fa2ea698098cda891c3923" exitCode=0
Sep 17 01:06:11.829361 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:11.829254    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6" event={"ID":"cd48a113-875b-47a7-a45b-1ae64154cf78","Type":"ContainerDied","Data":"863661966d87aeb54987f4ecb0f201a8bd322d4194fa2ea698098cda891c3923"}
Sep 17 01:06:12.282429 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:12.282396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:12.969586 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:12.969566    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"
Sep 17 01:06:13.055042 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:13.055013    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:13.107923 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:13.107863    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48a113-875b-47a7-a45b-1ae64154cf78-maas-api-service-ca\") pod \"cd48a113-875b-47a7-a45b-1ae64154cf78\" (UID: \"cd48a113-875b-47a7-a45b-1ae64154cf78\") "
Sep 17 01:06:13.108252 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:13.108219    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd48a113-875b-47a7-a45b-1ae64154cf78-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd48a113-875b-47a7-a45b-1ae64154cf78" (UID: "cd48a113-875b-47a7-a45b-1ae64154cf78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:06:13.208830 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:13.208801    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48a113-875b-47a7-a45b-1ae64154cf78-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:13.840820 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:13.840795    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"
Sep 17 01:06:13.840982 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:13.840837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6" event={"ID":"cd48a113-875b-47a7-a45b-1ae64154cf78","Type":"ContainerDied","Data":"d45ea3e3e0fb7cc04f0c7806d7c6e793b6a6f7f6a28e14bb6a1e5e895a0a2bec"}
Sep 17 01:06:13.840982 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:13.840882    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d45ea3e3e0fb7cc04f0c7806d7c6e793b6a6f7f6a28e14bb6a1e5e895a0a2bec"
Sep 17 01:06:20.378741 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.378700    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:20.402458 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.402424    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"]
Sep 17 01:06:20.403331 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.403294    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8" podUID="dde081c9-29fd-4a8e-935c-03f5918f2fa3" containerName="payload-processing" containerID="cri-o://e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b" gracePeriod=30
Sep 17 01:06:20.406334 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.406299    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"]
Sep 17 01:06:20.406711 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.406687    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l" podUID="9e2bbc28-1888-4b4e-ba4a-755428aba06b" containerName="payload-pre-processing" containerID="cri-o://ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2" gracePeriod=30
Sep 17 01:06:20.450179 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.450115    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:20.512688 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.512662    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:20.547165 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.547118    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"]
Sep 17 01:06:20.550445 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.550411    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c13796-rggd6"]
Sep 17 01:06:20.579377 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.579262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:20.592727 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.592688    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd48a113-875b-47a7-a45b-1ae64154cf78" path="/var/lib/kubelet/pods/cd48a113-875b-47a7-a45b-1ae64154cf78/volumes"
Sep 17 01:06:20.704888 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.704868    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:06:20.707624 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.707606    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:06:20.771578 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.771550    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2bbc28-1888-4b4e-ba4a-755428aba06b-plugins-config-volume\") pod \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\" (UID: \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\") "
Sep 17 01:06:20.771726 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.771630    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmn96\" (UniqueName: \"kubernetes.io/projected/9e2bbc28-1888-4b4e-ba4a-755428aba06b-kube-api-access-kmn96\") pod \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\" (UID: \"9e2bbc28-1888-4b4e-ba4a-755428aba06b\") "
Sep 17 01:06:20.771971 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.771949    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2bbc28-1888-4b4e-ba4a-755428aba06b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e2bbc28-1888-4b4e-ba4a-755428aba06b" (UID: "9e2bbc28-1888-4b4e-ba4a-755428aba06b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:06:20.773880 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.773849    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2bbc28-1888-4b4e-ba4a-755428aba06b-kube-api-access-kmn96" (OuterVolumeSpecName: "kube-api-access-kmn96") pod "9e2bbc28-1888-4b4e-ba4a-755428aba06b" (UID: "9e2bbc28-1888-4b4e-ba4a-755428aba06b"). InnerVolumeSpecName "kube-api-access-kmn96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:06:20.872954 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.872926    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde081c9-29fd-4a8e-935c-03f5918f2fa3-plugins-config-volume\") pod \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\" (UID: \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\") "
Sep 17 01:06:20.873097 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.873081    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkmtb\" (UniqueName: \"kubernetes.io/projected/dde081c9-29fd-4a8e-935c-03f5918f2fa3-kube-api-access-fkmtb\") pod \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\" (UID: \"dde081c9-29fd-4a8e-935c-03f5918f2fa3\") "
Sep 17 01:06:20.873322 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.873301    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde081c9-29fd-4a8e-935c-03f5918f2fa3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dde081c9-29fd-4a8e-935c-03f5918f2fa3" (UID: "dde081c9-29fd-4a8e-935c-03f5918f2fa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:06:20.873391 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.873323    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmn96\" (UniqueName: \"kubernetes.io/projected/9e2bbc28-1888-4b4e-ba4a-755428aba06b-kube-api-access-kmn96\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:20.873391 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.873342    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2bbc28-1888-4b4e-ba4a-755428aba06b-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:20.875123 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.875101    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde081c9-29fd-4a8e-935c-03f5918f2fa3-kube-api-access-fkmtb" (OuterVolumeSpecName: "kube-api-access-fkmtb") pod "dde081c9-29fd-4a8e-935c-03f5918f2fa3" (UID: "dde081c9-29fd-4a8e-935c-03f5918f2fa3"). InnerVolumeSpecName "kube-api-access-fkmtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:06:20.879220 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.879197    2576 generic.go:356] "Generic (PLEG): container finished" podID="dde081c9-29fd-4a8e-935c-03f5918f2fa3" containerID="e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b" exitCode=0
Sep 17 01:06:20.879328 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.879270    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"
Sep 17 01:06:20.879328 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.879280    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8" event={"ID":"dde081c9-29fd-4a8e-935c-03f5918f2fa3","Type":"ContainerDied","Data":"e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b"}
Sep 17 01:06:20.879437 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.879330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8" event={"ID":"dde081c9-29fd-4a8e-935c-03f5918f2fa3","Type":"ContainerDied","Data":"9b01633e8297a288b367be6fdcd7a4fd77e241cb8ce3609bda104cf944ec5b08"}
Sep 17 01:06:20.879437 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.879359    2576 scope.go:117] "RemoveContainer" containerID="e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b"
Sep 17 01:06:20.880650 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.880628    2576 generic.go:356] "Generic (PLEG): container finished" podID="9e2bbc28-1888-4b4e-ba4a-755428aba06b" containerID="ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2" exitCode=0
Sep 17 01:06:20.880756 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.880668    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"
Sep 17 01:06:20.880756 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.880682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l" event={"ID":"9e2bbc28-1888-4b4e-ba4a-755428aba06b","Type":"ContainerDied","Data":"ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2"}
Sep 17 01:06:20.880756 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.880720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l" event={"ID":"9e2bbc28-1888-4b4e-ba4a-755428aba06b","Type":"ContainerDied","Data":"43d1a8d24e9bfe190f0679992600f769558c17961fbab582fe7f6ce52b287f74"}
Sep 17 01:06:20.889917 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.889898    2576 scope.go:117] "RemoveContainer" containerID="e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b"
Sep 17 01:06:20.890251 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:06:20.890217    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b\": container with ID starting with e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b not found: ID does not exist" containerID="e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b"
Sep 17 01:06:20.890319 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.890255    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b"} err="failed to get container status \"e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b\": rpc error: code = NotFound desc = could not find container \"e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b\": container with ID starting with e680d9b7aa70def2034682d3e6c983d01e94723286e28c7f112cac960a20dc1b not found: ID does not exist"
Sep 17 01:06:20.890319 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.890272    2576 scope.go:117] "RemoveContainer" containerID="ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2"
Sep 17 01:06:20.897997 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.897982    2576 scope.go:117] "RemoveContainer" containerID="ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2"
Sep 17 01:06:20.898278 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:06:20.898259    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2\": container with ID starting with ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2 not found: ID does not exist" containerID="ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2"
Sep 17 01:06:20.898344 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.898284    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2"} err="failed to get container status \"ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2\": rpc error: code = NotFound desc = could not find container \"ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2\": container with ID starting with ca410904c5e18edfd42c21f0ebbf5e2710385a8f90b3cbf5e9bcc3ebe8012db2 not found: ID does not exist"
Sep 17 01:06:20.907420 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.907400    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"]
Sep 17 01:06:20.909290 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.909273    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c13796-78f4cf88df-snhc8"]
Sep 17 01:06:20.917343 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.917324    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"]
Sep 17 01:06:20.922993 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.922973    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c13796-774c4d9dc8-cbg5l"]
Sep 17 01:06:20.925012 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.924995    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:20.974599 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.974539    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkmtb\" (UniqueName: \"kubernetes.io/projected/dde081c9-29fd-4a8e-935c-03f5918f2fa3-kube-api-access-fkmtb\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:20.974599 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.974591    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde081c9-29fd-4a8e-935c-03f5918f2fa3-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:20.984185 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:20.984160    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:21.645220 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:21.645194    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:22.350481 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:22.350445    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:22.588127 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:22.588089    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e2bbc28-1888-4b4e-ba4a-755428aba06b" path="/var/lib/kubelet/pods/9e2bbc28-1888-4b4e-ba4a-755428aba06b/volumes"
Sep 17 01:06:22.588572 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:22.588552    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dde081c9-29fd-4a8e-935c-03f5918f2fa3" path="/var/lib/kubelet/pods/dde081c9-29fd-4a8e-935c-03f5918f2fa3/volumes"
Sep 17 01:06:26.504397 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:26.504369    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:26.613989 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:26.613949    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:26.756694 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:26.756608    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"]
Sep 17 01:06:26.756881 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:26.756856    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy" containerID="cri-o://b8f6e9b0da340ce0cae806b6b711d306a665bae886eb4e3b4c73855f6cc8c486" gracePeriod=30
Sep 17 01:06:26.775726 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:26.775703    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:27.123427 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123392    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"]
Sep 17 01:06:27.123788 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123776    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd48a113-875b-47a7-a45b-1ae64154cf78" containerName="revoke-keys"
Sep 17 01:06:27.123833 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123791    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd48a113-875b-47a7-a45b-1ae64154cf78" containerName="revoke-keys"
Sep 17 01:06:27.123833 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123806    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e2bbc28-1888-4b4e-ba4a-755428aba06b" containerName="payload-pre-processing"
Sep 17 01:06:27.123833 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123811    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e2bbc28-1888-4b4e-ba4a-755428aba06b" containerName="payload-pre-processing"
Sep 17 01:06:27.123833 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123818    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dde081c9-29fd-4a8e-935c-03f5918f2fa3" containerName="payload-processing"
Sep 17 01:06:27.123833 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123824    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dde081c9-29fd-4a8e-935c-03f5918f2fa3" containerName="payload-processing"
Sep 17 01:06:27.123980 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123884    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e2bbc28-1888-4b4e-ba4a-755428aba06b" containerName="payload-pre-processing"
Sep 17 01:06:27.123980 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123893    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dde081c9-29fd-4a8e-935c-03f5918f2fa3" containerName="payload-processing"
Sep 17 01:06:27.123980 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.123902    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd48a113-875b-47a7-a45b-1ae64154cf78" containerName="revoke-keys"
Sep 17 01:06:27.128346 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.128328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"
Sep 17 01:06:27.130742 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.130716    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 01:06:27.133277 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.133253    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"]
Sep 17 01:06:27.231097 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.231067    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c10f1e-a44b-4b82-a637-d0e41fe1f362-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl\" (UID: \"90c10f1e-a44b-4b82-a637-d0e41fe1f362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"
Sep 17 01:06:27.260731 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.260711    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:27.332234 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.332184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c10f1e-a44b-4b82-a637-d0e41fe1f362-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl\" (UID: \"90c10f1e-a44b-4b82-a637-d0e41fe1f362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"
Sep 17 01:06:27.333156 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.333110    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c10f1e-a44b-4b82-a637-d0e41fe1f362-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl\" (UID: \"90c10f1e-a44b-4b82-a637-d0e41fe1f362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"
Sep 17 01:06:27.441847 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.441783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"
Sep 17 01:06:27.714240 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.714215    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"]
Sep 17 01:06:27.716580 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:06:27.716542    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c10f1e_a44b_4b82_a637_d0e41fe1f362.slice/crio-368f6a692c52c709e0d0f1792cc9beb481d96748aba32a4ba48a21ad14e0295a WatchSource:0}: Error finding container 368f6a692c52c709e0d0f1792cc9beb481d96748aba32a4ba48a21ad14e0295a: Status 404 returned error can't find the container with id 368f6a692c52c709e0d0f1792cc9beb481d96748aba32a4ba48a21ad14e0295a
Sep 17 01:06:27.923132 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.923104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl" event={"ID":"90c10f1e-a44b-4b82-a637-d0e41fe1f362","Type":"ContainerStarted","Data":"368f6a692c52c709e0d0f1792cc9beb481d96748aba32a4ba48a21ad14e0295a"}
Sep 17 01:06:27.999823 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:27.999764    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:28.751753 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:28.751672    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:28.930156 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:28.930118    2576 generic.go:356] "Generic (PLEG): container finished" podID="90c10f1e-a44b-4b82-a637-d0e41fe1f362" containerID="47db192626db65bd05c05ccbcc74e5fe6fe1c97d68027d379a422fcfd2d876d3" exitCode=0
Sep 17 01:06:28.930349 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:28.930201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl" event={"ID":"90c10f1e-a44b-4b82-a637-d0e41fe1f362","Type":"ContainerDied","Data":"47db192626db65bd05c05ccbcc74e5fe6fe1c97d68027d379a422fcfd2d876d3"}
Sep 17 01:06:29.443774 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:29.443741    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:30.071965 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:30.071936    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"
Sep 17 01:06:30.256425 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:30.256355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c10f1e-a44b-4b82-a637-d0e41fe1f362-maas-api-service-ca\") pod \"90c10f1e-a44b-4b82-a637-d0e41fe1f362\" (UID: \"90c10f1e-a44b-4b82-a637-d0e41fe1f362\") "
Sep 17 01:06:30.256754 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:30.256733    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c10f1e-a44b-4b82-a637-d0e41fe1f362-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90c10f1e-a44b-4b82-a637-d0e41fe1f362" (UID: "90c10f1e-a44b-4b82-a637-d0e41fe1f362"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:06:30.357240 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:30.357217    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c10f1e-a44b-4b82-a637-d0e41fe1f362-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:30.942771 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:30.942692    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"
Sep 17 01:06:30.942914 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:30.942692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl" event={"ID":"90c10f1e-a44b-4b82-a637-d0e41fe1f362","Type":"ContainerDied","Data":"368f6a692c52c709e0d0f1792cc9beb481d96748aba32a4ba48a21ad14e0295a"}
Sep 17 01:06:30.942914 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:30.942815    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="368f6a692c52c709e0d0f1792cc9beb481d96748aba32a4ba48a21ad14e0295a"
Sep 17 01:06:31.817501 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:31.817476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:31.952123 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:31.952091    2576 generic.go:356] "Generic (PLEG): container finished" podID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerID="b8f6e9b0da340ce0cae806b6b711d306a665bae886eb4e3b4c73855f6cc8c486" exitCode=0
Sep 17 01:06:31.952279 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:31.952181    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" event={"ID":"6219c715-95cd-49c0-93ee-9351cfe223ad","Type":"ContainerDied","Data":"b8f6e9b0da340ce0cae806b6b711d306a665bae886eb4e3b4c73855f6cc8c486"}
Sep 17 01:06:31.952279 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:31.952241    2576 scope.go:117] "RemoveContainer" containerID="3ef0f616047a95893bd4160c3905a4496615e08a94e631a9ff8a461897b27672"
Sep 17 01:06:32.011869 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.011848    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:06:32.173159 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173127    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6219c715-95cd-49c0-93ee-9351cfe223ad-istiod-ca-cert\") pod \"6219c715-95cd-49c0-93ee-9351cfe223ad\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") "
Sep 17 01:06:32.173332 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173194    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-token\") pod \"6219c715-95cd-49c0-93ee-9351cfe223ad\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") "
Sep 17 01:06:32.173332 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173259    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-credential-socket\") pod \"6219c715-95cd-49c0-93ee-9351cfe223ad\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") "
Sep 17 01:06:32.173332 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173306    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-certs\") pod \"6219c715-95cd-49c0-93ee-9351cfe223ad\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") "
Sep 17 01:06:32.173505 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173338    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-data\") pod \"6219c715-95cd-49c0-93ee-9351cfe223ad\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") "
Sep 17 01:06:32.173505 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173370    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-envoy\") pod \"6219c715-95cd-49c0-93ee-9351cfe223ad\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") "
Sep 17 01:06:32.173505 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173396    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8hn7\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-kube-api-access-z8hn7\") pod \"6219c715-95cd-49c0-93ee-9351cfe223ad\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") "
Sep 17 01:06:32.173505 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173428    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-socket\") pod \"6219c715-95cd-49c0-93ee-9351cfe223ad\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") "
Sep 17 01:06:32.173505 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173464    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-podinfo\") pod \"6219c715-95cd-49c0-93ee-9351cfe223ad\" (UID: \"6219c715-95cd-49c0-93ee-9351cfe223ad\") "
Sep 17 01:06:32.173756 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173584    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "6219c715-95cd-49c0-93ee-9351cfe223ad" (UID: "6219c715-95cd-49c0-93ee-9351cfe223ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:32.173756 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173593    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6219c715-95cd-49c0-93ee-9351cfe223ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6219c715-95cd-49c0-93ee-9351cfe223ad" (UID: "6219c715-95cd-49c0-93ee-9351cfe223ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:06:32.173756 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173594    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6219c715-95cd-49c0-93ee-9351cfe223ad" (UID: "6219c715-95cd-49c0-93ee-9351cfe223ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:32.173756 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173610    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6219c715-95cd-49c0-93ee-9351cfe223ad" (UID: "6219c715-95cd-49c0-93ee-9351cfe223ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:32.173955 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173818    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6219c715-95cd-49c0-93ee-9351cfe223ad" (UID: "6219c715-95cd-49c0-93ee-9351cfe223ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:32.173955 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173844    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:32.173955 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173862    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:32.173955 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173871    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:32.173955 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.173880    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6219c715-95cd-49c0-93ee-9351cfe223ad-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:32.175809 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.175785    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6219c715-95cd-49c0-93ee-9351cfe223ad" (UID: "6219c715-95cd-49c0-93ee-9351cfe223ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 01:06:32.175915 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.175822    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-kube-api-access-z8hn7" (OuterVolumeSpecName: "kube-api-access-z8hn7") pod "6219c715-95cd-49c0-93ee-9351cfe223ad" (UID: "6219c715-95cd-49c0-93ee-9351cfe223ad"). InnerVolumeSpecName "kube-api-access-z8hn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:06:32.176040 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.176020    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6219c715-95cd-49c0-93ee-9351cfe223ad" (UID: "6219c715-95cd-49c0-93ee-9351cfe223ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:32.176097 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.176033    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "6219c715-95cd-49c0-93ee-9351cfe223ad" (UID: "6219c715-95cd-49c0-93ee-9351cfe223ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:06:32.274994 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.274969    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:32.274994 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.274990    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:32.274994 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.274998    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8hn7\" (UniqueName: \"kubernetes.io/projected/6219c715-95cd-49c0-93ee-9351cfe223ad-kube-api-access-z8hn7\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:32.275172 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.275007    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6219c715-95cd-49c0-93ee-9351cfe223ad-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:32.275172 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.275016    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6219c715-95cd-49c0-93ee-9351cfe223ad-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:32.958496 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.958425    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"
Sep 17 01:06:32.958496 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.958465    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m" event={"ID":"6219c715-95cd-49c0-93ee-9351cfe223ad","Type":"ContainerDied","Data":"f1746babf548ff9750c81bfaa64e01f5b31856f7702355215b5f381d4c20eea0"}
Sep 17 01:06:32.958998 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.958521    2576 scope.go:117] "RemoveContainer" containerID="b8f6e9b0da340ce0cae806b6b711d306a665bae886eb4e3b4c73855f6cc8c486"
Sep 17 01:06:32.975159 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.975118    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"]
Sep 17 01:06:32.978504 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:32.978484    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c13796-openshift-default-6bbddbd97d-dzh5m"]
Sep 17 01:06:34.587710 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:34.587673    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" path="/var/lib/kubelet/pods/6219c715-95cd-49c0-93ee-9351cfe223ad/volumes"
Sep 17 01:06:37.572657 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.572624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:37.601054 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.601017    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"]
Sep 17 01:06:37.601356 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.601326    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj" podUID="a95a5e91-97b0-4ac8-86e8-d40f8fde0175" containerName="payload-processing" containerID="cri-o://7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea" gracePeriod=30
Sep 17 01:06:37.641372 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.641341    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:37.744051 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.744012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:37.756935 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.756905    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"]
Sep 17 01:06:37.758966 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.758946    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d2326d-r6qsl"]
Sep 17 01:06:37.797674 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.797616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:37.869167 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.869131    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 01:06:37.915048 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.915025    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9drv2\" (UniqueName: \"kubernetes.io/projected/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-kube-api-access-9drv2\") pod \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\" (UID: \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\") "
Sep 17 01:06:37.915203 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.915083    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-plugins-config-volume\") pod \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\" (UID: \"a95a5e91-97b0-4ac8-86e8-d40f8fde0175\") "
Sep 17 01:06:37.915458 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.915437    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a95a5e91-97b0-4ac8-86e8-d40f8fde0175" (UID: "a95a5e91-97b0-4ac8-86e8-d40f8fde0175"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:06:37.917240 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.917215    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-kube-api-access-9drv2" (OuterVolumeSpecName: "kube-api-access-9drv2") pod "a95a5e91-97b0-4ac8-86e8-d40f8fde0175" (UID: "a95a5e91-97b0-4ac8-86e8-d40f8fde0175"). InnerVolumeSpecName "kube-api-access-9drv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:06:37.991645 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.991610    2576 generic.go:356] "Generic (PLEG): container finished" podID="a95a5e91-97b0-4ac8-86e8-d40f8fde0175" containerID="7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea" exitCode=0
Sep 17 01:06:37.991747 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.991654    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"
Sep 17 01:06:37.991747 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.991653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj" event={"ID":"a95a5e91-97b0-4ac8-86e8-d40f8fde0175","Type":"ContainerDied","Data":"7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea"}
Sep 17 01:06:37.991747 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.991707    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj" event={"ID":"a95a5e91-97b0-4ac8-86e8-d40f8fde0175","Type":"ContainerDied","Data":"92e1802f8d592d7675c518aeaa5469322946e1fd564fcb3186563eb38abb7647"}
Sep 17 01:06:37.991747 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:37.991729    2576 scope.go:117] "RemoveContainer" containerID="7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea"
Sep 17 01:06:38.000694 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.000678    2576 scope.go:117] "RemoveContainer" containerID="7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea"
Sep 17 01:06:38.000949 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:06:38.000929    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea\": container with ID starting with 7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea not found: ID does not exist" containerID="7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea"
Sep 17 01:06:38.001011 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.000958    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea"} err="failed to get container status \"7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea\": rpc error: code = NotFound desc = could not find container \"7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea\": container with ID starting with 7bef83c018c4acfc9dfc9e20f5b36e4f7a0a5c278da966592dbacd0827006bea not found: ID does not exist"
Sep 17 01:06:38.011815 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.011793    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"]
Sep 17 01:06:38.015394 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.015374    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d2326d-8649f769-hpgtj"]
Sep 17 01:06:38.015722 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.015706    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9drv2\" (UniqueName: \"kubernetes.io/projected/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-kube-api-access-9drv2\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:38.015765 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.015725    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95a5e91-97b0-4ac8-86e8-d40f8fde0175-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:38.132118 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.132036    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:38.232905 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.232881    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:38.587783 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.587753    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c10f1e-a44b-4b82-a637-d0e41fe1f362" path="/var/lib/kubelet/pods/90c10f1e-a44b-4b82-a637-d0e41fe1f362/volumes"
Sep 17 01:06:38.588132 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.588098    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a95a5e91-97b0-4ac8-86e8-d40f8fde0175" path="/var/lib/kubelet/pods/a95a5e91-97b0-4ac8-86e8-d40f8fde0175/volumes"
Sep 17 01:06:38.970978 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:38.970864    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:43.981273 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:43.981211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:44.044106 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.044071    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:44.208869 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.208830    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"]
Sep 17 01:06:44.209106 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.209078    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy" containerID="cri-o://c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b" gracePeriod=30
Sep 17 01:06:44.233156 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.233085    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:44.580707 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.580676    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"]
Sep 17 01:06:44.581043 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581030    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95a5e91-97b0-4ac8-86e8-d40f8fde0175" containerName="payload-processing"
Sep 17 01:06:44.581102 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581045    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95a5e91-97b0-4ac8-86e8-d40f8fde0175" containerName="payload-processing"
Sep 17 01:06:44.581102 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581062    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy"
Sep 17 01:06:44.581102 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581067    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy"
Sep 17 01:06:44.581102 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581073    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c10f1e-a44b-4b82-a637-d0e41fe1f362" containerName="revoke-keys"
Sep 17 01:06:44.581102 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581079    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c10f1e-a44b-4b82-a637-d0e41fe1f362" containerName="revoke-keys"
Sep 17 01:06:44.581102 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581097    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy"
Sep 17 01:06:44.581102 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581102    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy"
Sep 17 01:06:44.581347 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581181    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c10f1e-a44b-4b82-a637-d0e41fe1f362" containerName="revoke-keys"
Sep 17 01:06:44.581347 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581193    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a95a5e91-97b0-4ac8-86e8-d40f8fde0175" containerName="payload-processing"
Sep 17 01:06:44.581347 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.581200    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy"
Sep 17 01:06:44.584456 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.584433    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"
Sep 17 01:06:44.586748 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.586731    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f2hd4\""
Sep 17 01:06:44.591568 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.591548    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"]
Sep 17 01:06:44.671881 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.671852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350d8ac6-7f68-42e3-85a2-54ae1cbca681-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v\" (UID: \"350d8ac6-7f68-42e3-85a2-54ae1cbca681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"
Sep 17 01:06:44.744845 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.744818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:44.772942 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.772916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350d8ac6-7f68-42e3-85a2-54ae1cbca681-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v\" (UID: \"350d8ac6-7f68-42e3-85a2-54ae1cbca681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"
Sep 17 01:06:44.773543 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.773525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350d8ac6-7f68-42e3-85a2-54ae1cbca681-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v\" (UID: \"350d8ac6-7f68-42e3-85a2-54ae1cbca681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"
Sep 17 01:06:44.897231 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:44.897134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"
Sep 17 01:06:45.168853 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:45.168830    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"]
Sep 17 01:06:45.171044 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:06:45.171011    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350d8ac6_7f68_42e3_85a2_54ae1cbca681.slice/crio-46074817ac71553db90dc035c3dfd204c20810fa34920d7267ef57b8e1b26007 WatchSource:0}: Error finding container 46074817ac71553db90dc035c3dfd204c20810fa34920d7267ef57b8e1b26007: Status 404 returned error can't find the container with id 46074817ac71553db90dc035c3dfd204c20810fa34920d7267ef57b8e1b26007
Sep 17 01:06:45.543738 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:45.543709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:46.038002 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:46.037962    2576 generic.go:356] "Generic (PLEG): container finished" podID="350d8ac6-7f68-42e3-85a2-54ae1cbca681" containerID="b3f6f5a83b4ffd4bed2dbb3b4d8411cba43487bd87c14e1d8ba418ccf18fde2b" exitCode=0
Sep 17 01:06:46.038186 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:46.038055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v" event={"ID":"350d8ac6-7f68-42e3-85a2-54ae1cbca681","Type":"ContainerDied","Data":"b3f6f5a83b4ffd4bed2dbb3b4d8411cba43487bd87c14e1d8ba418ccf18fde2b"}
Sep 17 01:06:46.038186 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:46.038114    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v" event={"ID":"350d8ac6-7f68-42e3-85a2-54ae1cbca681","Type":"ContainerStarted","Data":"46074817ac71553db90dc035c3dfd204c20810fa34920d7267ef57b8e1b26007"}
Sep 17 01:06:46.306201 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:46.306119    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:47.179759 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:47.179736    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"
Sep 17 01:06:47.295004 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:47.294931    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350d8ac6-7f68-42e3-85a2-54ae1cbca681-maas-api-service-ca\") pod \"350d8ac6-7f68-42e3-85a2-54ae1cbca681\" (UID: \"350d8ac6-7f68-42e3-85a2-54ae1cbca681\") "
Sep 17 01:06:47.295317 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:47.295296    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350d8ac6-7f68-42e3-85a2-54ae1cbca681-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "350d8ac6-7f68-42e3-85a2-54ae1cbca681" (UID: "350d8ac6-7f68-42e3-85a2-54ae1cbca681"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:06:47.396406 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:47.396365    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/350d8ac6-7f68-42e3-85a2-54ae1cbca681-maas-api-service-ca\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:48.051139 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:48.051102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v" event={"ID":"350d8ac6-7f68-42e3-85a2-54ae1cbca681","Type":"ContainerDied","Data":"46074817ac71553db90dc035c3dfd204c20810fa34920d7267ef57b8e1b26007"}
Sep 17 01:06:48.051381 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:48.051164    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"
Sep 17 01:06:48.051381 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:48.051180    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46074817ac71553db90dc035c3dfd204c20810fa34920d7267ef57b8e1b26007"
Sep 17 01:06:49.462561 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.462540    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 01:06:49.513320 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513257    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-credential-socket\") pod \"5f50785f-463a-4c64-9447-c635eb575d94\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") "
Sep 17 01:06:49.513320 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513291    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-envoy\") pod \"5f50785f-463a-4c64-9447-c635eb575d94\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") "
Sep 17 01:06:49.513320 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513311    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-data\") pod \"5f50785f-463a-4c64-9447-c635eb575d94\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") "
Sep 17 01:06:49.513536 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513357    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f50785f-463a-4c64-9447-c635eb575d94-istio-podinfo\") pod \"5f50785f-463a-4c64-9447-c635eb575d94\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") "
Sep 17 01:06:49.513536 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513385    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-socket\") pod \"5f50785f-463a-4c64-9447-c635eb575d94\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") "
Sep 17 01:06:49.513536 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-certs\") pod \"5f50785f-463a-4c64-9447-c635eb575d94\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") "
Sep 17 01:06:49.513536 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513419    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f50785f-463a-4c64-9447-c635eb575d94-istiod-ca-cert\") pod \"5f50785f-463a-4c64-9447-c635eb575d94\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") "
Sep 17 01:06:49.513536 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513475    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-556fc\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-kube-api-access-556fc\") pod \"5f50785f-463a-4c64-9447-c635eb575d94\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") "
Sep 17 01:06:49.513536 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513506    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-istio-token\") pod \"5f50785f-463a-4c64-9447-c635eb575d94\" (UID: \"5f50785f-463a-4c64-9447-c635eb575d94\") "
Sep 17 01:06:49.513536 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f50785f-463a-4c64-9447-c635eb575d94" (UID: "5f50785f-463a-4c64-9447-c635eb575d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:49.513892 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513711    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f50785f-463a-4c64-9447-c635eb575d94" (UID: "5f50785f-463a-4c64-9447-c635eb575d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:49.513892 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513752    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f50785f-463a-4c64-9447-c635eb575d94" (UID: "5f50785f-463a-4c64-9447-c635eb575d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:49.513892 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513716    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f50785f-463a-4c64-9447-c635eb575d94" (UID: "5f50785f-463a-4c64-9447-c635eb575d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:49.513892 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513853    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:49.513892 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513854    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f50785f-463a-4c64-9447-c635eb575d94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f50785f-463a-4c64-9447-c635eb575d94" (UID: "5f50785f-463a-4c64-9447-c635eb575d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:06:49.513892 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513868    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-workload-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:49.513892 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513877    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-credential-socket\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:49.513892 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.513886    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-data\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:49.515773 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.515747    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f50785f-463a-4c64-9447-c635eb575d94" (UID: "5f50785f-463a-4c64-9447-c635eb575d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:06:49.515907 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.515780    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f50785f-463a-4c64-9447-c635eb575d94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f50785f-463a-4c64-9447-c635eb575d94" (UID: "5f50785f-463a-4c64-9447-c635eb575d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 01:06:49.515907 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.515782    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-kube-api-access-556fc" (OuterVolumeSpecName: "kube-api-access-556fc") pod "5f50785f-463a-4c64-9447-c635eb575d94" (UID: "5f50785f-463a-4c64-9447-c635eb575d94"). InnerVolumeSpecName "kube-api-access-556fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:06:49.515907 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.515838    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f50785f-463a-4c64-9447-c635eb575d94" (UID: "5f50785f-463a-4c64-9447-c635eb575d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:06:49.615317 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.615293    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f50785f-463a-4c64-9447-c635eb575d94-istio-podinfo\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:49.615317 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.615314    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f50785f-463a-4c64-9447-c635eb575d94-istiod-ca-cert\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:49.615471 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.615325    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556fc\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-kube-api-access-556fc\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:49.615471 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.615333    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f50785f-463a-4c64-9447-c635eb575d94-istio-token\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:49.615471 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:49.615342    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f50785f-463a-4c64-9447-c635eb575d94-istio-envoy\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:50.063149 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.063119    2576 generic.go:356] "Generic (PLEG): container finished" podID="5f50785f-463a-4c64-9447-c635eb575d94" containerID="c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b" exitCode=0
Sep 17 01:06:50.063351 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.063178    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"
Sep 17 01:06:50.063351 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.063212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" event={"ID":"5f50785f-463a-4c64-9447-c635eb575d94","Type":"ContainerDied","Data":"c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b"}
Sep 17 01:06:50.063351 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.063275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8" event={"ID":"5f50785f-463a-4c64-9447-c635eb575d94","Type":"ContainerDied","Data":"a181965ee4e5eb9c8d0a81690d05f12364700966e68c2ee7da6a148c5d9cab21"}
Sep 17 01:06:50.063351 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.063307    2576 scope.go:117] "RemoveContainer" containerID="c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b"
Sep 17 01:06:50.073134 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.073120    2576 scope.go:117] "RemoveContainer" containerID="cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93"
Sep 17 01:06:50.081293 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.081275    2576 scope.go:117] "RemoveContainer" containerID="c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b"
Sep 17 01:06:50.081556 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:06:50.081538    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b\": container with ID starting with c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b not found: ID does not exist" containerID="c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b"
Sep 17 01:06:50.081604 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.081565    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b"} err="failed to get container status \"c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b\": rpc error: code = NotFound desc = could not find container \"c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b\": container with ID starting with c3463a689219b73345e60798f86ed0229c36f763e3f0564151d2335db393a26b not found: ID does not exist"
Sep 17 01:06:50.081604 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.081583    2576 scope.go:117] "RemoveContainer" containerID="cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93"
Sep 17 01:06:50.081833 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:06:50.081817    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93\": container with ID starting with cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93 not found: ID does not exist" containerID="cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93"
Sep 17 01:06:50.081887 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.081836    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93"} err="failed to get container status \"cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93\": rpc error: code = NotFound desc = could not find container \"cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93\": container with ID starting with cb054c1625f1952d2f5c0d36903db40f841f1b88d8631db22d7b4227a8d4ae93 not found: ID does not exist"
Sep 17 01:06:50.085257 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.085238    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"]
Sep 17 01:06:50.088259 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.088240    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d2326d-openshift-default-74ff559bf4-7wjd8"]
Sep 17 01:06:50.588002 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:50.587968    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f50785f-463a-4c64-9447-c635eb575d94" path="/var/lib/kubelet/pods/5f50785f-463a-4c64-9447-c635eb575d94/volumes"
Sep 17 01:06:55.031328 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.031288    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:55.040860 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.040802    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2"]
Sep 17 01:06:55.042759 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.042730    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcd112-29826780-t5wg2"]
Sep 17 01:06:55.069011 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.068977    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"]
Sep 17 01:06:55.069311 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.069270    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf" podUID="2fb085f8-5546-4b70-9293-1b3e437745b3" containerName="payload-pre-processing" containerID="cri-o://f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541" gracePeriod=30
Sep 17 01:06:55.131274 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.131250    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:55.214331 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.214299    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"]
Sep 17 01:06:55.217980 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.217955    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bcd112-7k58v"]
Sep 17 01:06:55.218203 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.218185    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:55.273012 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.272971    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:55.323831 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.323810    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 01:06:55.463436 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.463406    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb085f8-5546-4b70-9293-1b3e437745b3-plugins-config-volume\") pod \"2fb085f8-5546-4b70-9293-1b3e437745b3\" (UID: \"2fb085f8-5546-4b70-9293-1b3e437745b3\") "
Sep 17 01:06:55.463613 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.463459    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j86z5\" (UniqueName: \"kubernetes.io/projected/2fb085f8-5546-4b70-9293-1b3e437745b3-kube-api-access-j86z5\") pod \"2fb085f8-5546-4b70-9293-1b3e437745b3\" (UID: \"2fb085f8-5546-4b70-9293-1b3e437745b3\") "
Sep 17 01:06:55.463822 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.463798    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb085f8-5546-4b70-9293-1b3e437745b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2fb085f8-5546-4b70-9293-1b3e437745b3" (UID: "2fb085f8-5546-4b70-9293-1b3e437745b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 01:06:55.465673 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.465651    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb085f8-5546-4b70-9293-1b3e437745b3-kube-api-access-j86z5" (OuterVolumeSpecName: "kube-api-access-j86z5") pod "2fb085f8-5546-4b70-9293-1b3e437745b3" (UID: "2fb085f8-5546-4b70-9293-1b3e437745b3"). InnerVolumeSpecName "kube-api-access-j86z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:06:55.564891 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.564828    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb085f8-5546-4b70-9293-1b3e437745b3-plugins-config-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:55.564891 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.564852    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86z5\" (UniqueName: \"kubernetes.io/projected/2fb085f8-5546-4b70-9293-1b3e437745b3-kube-api-access-j86z5\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:06:55.582420 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.582398    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:55.682272 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:55.682246    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:06:56.099694 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.099659    2576 generic.go:356] "Generic (PLEG): container finished" podID="2fb085f8-5546-4b70-9293-1b3e437745b3" containerID="f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541" exitCode=0
Sep 17 01:06:56.100089 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.099705    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"
Sep 17 01:06:56.100089 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.099713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf" event={"ID":"2fb085f8-5546-4b70-9293-1b3e437745b3","Type":"ContainerDied","Data":"f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541"}
Sep 17 01:06:56.100089 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.099787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf" event={"ID":"2fb085f8-5546-4b70-9293-1b3e437745b3","Type":"ContainerDied","Data":"b97483c69ebdaf954cf377eabb341deda4dc6ec51d679316fed645c6cc1c6c1a"}
Sep 17 01:06:56.100089 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.099815    2576 scope.go:117] "RemoveContainer" containerID="f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541"
Sep 17 01:06:56.111572 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.111553    2576 scope.go:117] "RemoveContainer" containerID="f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541"
Sep 17 01:06:56.111855 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:06:56.111836    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541\": container with ID starting with f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541 not found: ID does not exist" containerID="f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541"
Sep 17 01:06:56.111912 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.111864    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541"} err="failed to get container status \"f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541\": rpc error: code = NotFound desc = could not find container \"f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541\": container with ID starting with f97d6d8d1dabc3bd0cba59b573f06ab50ab43c6e99cb865b27505c45fa9e4541 not found: ID does not exist"
Sep 17 01:06:56.120562 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.120538    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"]
Sep 17 01:06:56.123803 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.123782    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcd112-b4dc47598-2lxpf"]
Sep 17 01:06:56.586941 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.586917    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb085f8-5546-4b70-9293-1b3e437745b3" path="/var/lib/kubelet/pods/2fb085f8-5546-4b70-9293-1b3e437745b3/volumes"
Sep 17 01:06:56.587262 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.587248    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="350d8ac6-7f68-42e3-85a2-54ae1cbca681" path="/var/lib/kubelet/pods/350d8ac6-7f68-42e3-85a2-54ae1cbca681/volumes"
Sep 17 01:06:56.587489 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:06:56.587478    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b8f5cb-df8e-4da8-a46b-6eadf4fb25a5" path="/var/lib/kubelet/pods/39b8f5cb-df8e-4da8-a46b-6eadf4fb25a5/volumes"
Sep 17 01:07:01.366132 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:07:01.366100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:07:01.422594 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:07:01.422560    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:07:01.627716 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:07:01.627639    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:07:05.124828 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:07:05.124794    2576 scope.go:117] "RemoveContainer" containerID="04e544e5195e34a4295b6226f430d55e1fbdb14b365dae15d654fa43edf89be9"
Sep 17 01:07:05.134584 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:07:05.134557    2576 scope.go:117] "RemoveContainer" containerID="99a89ae35b0ae894acb219440c55144bd9c3a3af04501760338a51058971b2c4"
Sep 17 01:07:05.143176 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:07:05.143138    2576 scope.go:117] "RemoveContainer" containerID="5a6edc125eae9df4eff5fad8aaed54ac5bf4caa3b83a0de4263494777d6904f0"
Sep 17 01:08:10.618531 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:08:10.618481    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:08:10.668843 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:08:10.668813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:08:27.912332 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:08:27.912296    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:08:29.466804 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:08:29.466772    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:09:44.617009 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.616979    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"]
Sep 17 01:09:44.617630 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.617174    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl" podUID="90330fe6-2e01-4c2c-a199-723ef4f20613" containerName="manager" containerID="cri-o://23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256" gracePeriod=10
Sep 17 01:09:44.872174 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.872100    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 01:09:44.965323 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.965296    2576 generic.go:356] "Generic (PLEG): container finished" podID="90330fe6-2e01-4c2c-a199-723ef4f20613" containerID="23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256" exitCode=0
Sep 17 01:09:44.965485 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.965336    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"
Sep 17 01:09:44.965485 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.965378    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl" event={"ID":"90330fe6-2e01-4c2c-a199-723ef4f20613","Type":"ContainerDied","Data":"23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256"}
Sep 17 01:09:44.965485 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.965423    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl" event={"ID":"90330fe6-2e01-4c2c-a199-723ef4f20613","Type":"ContainerDied","Data":"e7adda8de000bae78bcdb77ae737284d041e85dd77e0c2834804aaef5203d827"}
Sep 17 01:09:44.965485 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.965444    2576 scope.go:117] "RemoveContainer" containerID="23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256"
Sep 17 01:09:44.974337 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.974324    2576 scope.go:117] "RemoveContainer" containerID="23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256"
Sep 17 01:09:44.974608 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:09:44.974593    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256\": container with ID starting with 23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256 not found: ID does not exist" containerID="23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256"
Sep 17 01:09:44.974654 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.974618    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256"} err="failed to get container status \"23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256\": rpc error: code = NotFound desc = could not find container \"23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256\": container with ID starting with 23fd77a95eb032a0a8b3d5748f0530b84ff8a4d0c3a2607014a704bd67969256 not found: ID does not exist"
Sep 17 01:09:44.983376 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.983355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmlkh\" (UniqueName: \"kubernetes.io/projected/90330fe6-2e01-4c2c-a199-723ef4f20613-kube-api-access-xmlkh\") pod \"90330fe6-2e01-4c2c-a199-723ef4f20613\" (UID: \"90330fe6-2e01-4c2c-a199-723ef4f20613\") "
Sep 17 01:09:44.983456 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.983412    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90330fe6-2e01-4c2c-a199-723ef4f20613-extensions-socket-volume\") pod \"90330fe6-2e01-4c2c-a199-723ef4f20613\" (UID: \"90330fe6-2e01-4c2c-a199-723ef4f20613\") "
Sep 17 01:09:44.983752 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.983734    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90330fe6-2e01-4c2c-a199-723ef4f20613-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "90330fe6-2e01-4c2c-a199-723ef4f20613" (UID: "90330fe6-2e01-4c2c-a199-723ef4f20613"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:09:44.985395 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:44.985369    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90330fe6-2e01-4c2c-a199-723ef4f20613-kube-api-access-xmlkh" (OuterVolumeSpecName: "kube-api-access-xmlkh") pod "90330fe6-2e01-4c2c-a199-723ef4f20613" (UID: "90330fe6-2e01-4c2c-a199-723ef4f20613"). InnerVolumeSpecName "kube-api-access-xmlkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:09:45.084544 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:45.084523    2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90330fe6-2e01-4c2c-a199-723ef4f20613-extensions-socket-volume\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:09:45.084544 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:45.084544    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmlkh\" (UniqueName: \"kubernetes.io/projected/90330fe6-2e01-4c2c-a199-723ef4f20613-kube-api-access-xmlkh\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:09:45.287209 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:45.287187    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"]
Sep 17 01:09:45.291877 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:45.291858    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ln7xl"]
Sep 17 01:09:46.587238 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:09:46.587202    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90330fe6-2e01-4c2c-a199-723ef4f20613" path="/var/lib/kubelet/pods/90330fe6-2e01-4c2c-a199-723ef4f20613/volumes"
Sep 17 01:10:05.163987 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:05.163952    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 01:10:05.171016 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:05.170997    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tfff_116ae241-a23d-4780-ae66-a3ba9aee4499/ovn-acl-logging/0.log"
Sep 17 01:10:45.311847 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.311769    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"]
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312091    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90330fe6-2e01-4c2c-a199-723ef4f20613" containerName="manager"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312102    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90330fe6-2e01-4c2c-a199-723ef4f20613" containerName="manager"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312111    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="350d8ac6-7f68-42e3-85a2-54ae1cbca681" containerName="revoke-keys"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312116    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="350d8ac6-7f68-42e3-85a2-54ae1cbca681" containerName="revoke-keys"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312124    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312129    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312138    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312157    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312177    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb085f8-5546-4b70-9293-1b3e437745b3" containerName="payload-pre-processing"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312182    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb085f8-5546-4b70-9293-1b3e437745b3" containerName="payload-pre-processing"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312246    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312254    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6219c715-95cd-49c0-93ee-9351cfe223ad" containerName="istio-proxy"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312261    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90330fe6-2e01-4c2c-a199-723ef4f20613" containerName="manager"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312267    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="350d8ac6-7f68-42e3-85a2-54ae1cbca681" containerName="revoke-keys"
Sep 17 01:10:45.312335 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.312273    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb085f8-5546-4b70-9293-1b3e437745b3" containerName="payload-pre-processing"
Sep 17 01:10:45.315201 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.315184    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:10:45.317966 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.317941    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 01:10:45.318110 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.317944    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 01:10:45.318110 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.317993    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8qhv2\""
Sep 17 01:10:45.326273 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.326244    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"]
Sep 17 01:10:45.449590 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.449554    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95cdd121-9a20-49ab-a6c9-c9f1967064f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hbks6\" (UID: \"95cdd121-9a20-49ab-a6c9-c9f1967064f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:10:45.449771 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.449602    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwk6\" (UniqueName: \"kubernetes.io/projected/95cdd121-9a20-49ab-a6c9-c9f1967064f1-kube-api-access-brwk6\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hbks6\" (UID: \"95cdd121-9a20-49ab-a6c9-c9f1967064f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:10:45.550863 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.550834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95cdd121-9a20-49ab-a6c9-c9f1967064f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hbks6\" (UID: \"95cdd121-9a20-49ab-a6c9-c9f1967064f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:10:45.550987 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.550873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwk6\" (UniqueName: \"kubernetes.io/projected/95cdd121-9a20-49ab-a6c9-c9f1967064f1-kube-api-access-brwk6\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hbks6\" (UID: \"95cdd121-9a20-49ab-a6c9-c9f1967064f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:10:45.551241 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.551220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95cdd121-9a20-49ab-a6c9-c9f1967064f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hbks6\" (UID: \"95cdd121-9a20-49ab-a6c9-c9f1967064f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:10:45.559220 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.559191    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwk6\" (UniqueName: \"kubernetes.io/projected/95cdd121-9a20-49ab-a6c9-c9f1967064f1-kube-api-access-brwk6\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hbks6\" (UID: \"95cdd121-9a20-49ab-a6c9-c9f1967064f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:10:45.625218 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.625157    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:10:45.906869 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:45.906844    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"]
Sep 17 01:10:45.909394 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:10:45.909366    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cdd121_9a20_49ab_a6c9_c9f1967064f1.slice/crio-f73f7d1d31c8604fb4ba1ac995091aeeaca5f2c3ef465214735fed28d481089c WatchSource:0}: Error finding container f73f7d1d31c8604fb4ba1ac995091aeeaca5f2c3ef465214735fed28d481089c: Status 404 returned error can't find the container with id f73f7d1d31c8604fb4ba1ac995091aeeaca5f2c3ef465214735fed28d481089c
Sep 17 01:10:46.275776 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:46.275698    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6" event={"ID":"95cdd121-9a20-49ab-a6c9-c9f1967064f1","Type":"ContainerStarted","Data":"b78ccf07267a5bc7dbc92094acc4f52c336258ec6e1d1e57dbfbbc54b5945a6c"}
Sep 17 01:10:46.275776 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:46.275744    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6" event={"ID":"95cdd121-9a20-49ab-a6c9-c9f1967064f1","Type":"ContainerStarted","Data":"f73f7d1d31c8604fb4ba1ac995091aeeaca5f2c3ef465214735fed28d481089c"}
Sep 17 01:10:46.275974 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:46.275797    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:10:46.296990 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:46.296941    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6" podStartSLOduration=1.296928762 podStartE2EDuration="1.296928762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:10:46.294530549 +0000 UTC m=+2442.250126890" watchObservedRunningTime="2026-09-17 01:10:46.296928762 +0000 UTC m=+2442.252525103"
Sep 17 01:10:57.283578 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:10:57.283543    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hbks6"
Sep 17 01:11:05.215847 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:05.215814    2576 scope.go:117] "RemoveContainer" containerID="1365192d5346893688582a43890693daeda32d828d64f13d47c36d589b28a2e5"
Sep 17 01:11:07.797929 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:07.797889    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:11:20.192493 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.192453    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"]
Sep 17 01:11:20.192997 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.192981    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f50785f-463a-4c64-9447-c635eb575d94" containerName="istio-proxy"
Sep 17 01:11:20.196300 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.196279    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.198714 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.198697    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d97c19b1-kserve-self-signed-certs\""
Sep 17 01:11:20.204054 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.203877    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"]
Sep 17 01:11:20.220049 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.220027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-dshm\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.220194 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.220175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-home\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.220265 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.220202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-tmp-dir\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.220265 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.220218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d6e212-b218-4a3b-b472-9e35b034065e-tls-certs\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.220265 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.220245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-model-cache\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.220378 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.220301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fffx\" (UniqueName: \"kubernetes.io/projected/40d6e212-b218-4a3b-b472-9e35b034065e-kube-api-access-2fffx\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.225902 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.225871    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:11:20.321708 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.321685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-home\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.321815 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.321713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-tmp-dir\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.321815 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.321732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d6e212-b218-4a3b-b472-9e35b034065e-tls-certs\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.321815 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.321760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-model-cache\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.321815 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.321788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fffx\" (UniqueName: \"kubernetes.io/projected/40d6e212-b218-4a3b-b472-9e35b034065e-kube-api-access-2fffx\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.322040 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.321834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-dshm\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.322132 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.322114    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-home\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.322209 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.322193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-tmp-dir\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.322254 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.322222    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-model-cache\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.324121 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.324104    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-dshm\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.324318 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.324299    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d6e212-b218-4a3b-b472-9e35b034065e-tls-certs\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.329791 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.329767    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fffx\" (UniqueName: \"kubernetes.io/projected/40d6e212-b218-4a3b-b472-9e35b034065e-kube-api-access-2fffx\") pod \"e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") " pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.506839 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.506766    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:20.777175 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:20.777130    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"]
Sep 17 01:11:20.778901 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:11:20.778878    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d6e212_b218_4a3b_b472_9e35b034065e.slice/crio-ffbcf1c9e7d033fc1ea1dfd79509a9ddb02f5061eaa217570692c1a66118aaa9 WatchSource:0}: Error finding container ffbcf1c9e7d033fc1ea1dfd79509a9ddb02f5061eaa217570692c1a66118aaa9: Status 404 returned error can't find the container with id ffbcf1c9e7d033fc1ea1dfd79509a9ddb02f5061eaa217570692c1a66118aaa9
Sep 17 01:11:21.460803 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:21.460767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4" event={"ID":"40d6e212-b218-4a3b-b472-9e35b034065e","Type":"ContainerStarted","Data":"0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8"}
Sep 17 01:11:21.460803 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:21.460807    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4" event={"ID":"40d6e212-b218-4a3b-b472-9e35b034065e","Type":"ContainerStarted","Data":"ffbcf1c9e7d033fc1ea1dfd79509a9ddb02f5061eaa217570692c1a66118aaa9"}
Sep 17 01:11:21.476208 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:21.476139    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4" podStartSLOduration=1.476124116 podStartE2EDuration="1.476124116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:11:21.475540177 +0000 UTC m=+2477.431136520" watchObservedRunningTime="2026-09-17 01:11:21.476124116 +0000 UTC m=+2477.431720493"
Sep 17 01:11:30.507199 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:30.507160    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:30.507692 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:30.507220    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:30.519617 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:30.519594    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:31.523915 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:31.523881    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:11:33.189289 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.189245    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"]
Sep 17 01:11:33.193609 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.193586    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.196269 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.196249    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d97c19b1-kserve-self-signed-certs\""
Sep 17 01:11:33.202086 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.202065    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"]
Sep 17 01:11:33.220088 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.220048    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:11:33.335054 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.335028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfhgv\" (UniqueName: \"kubernetes.io/projected/f94a4eed-68eb-4de6-b8da-96d486d8db2c-kube-api-access-dfhgv\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.335196 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.335074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-home\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.335196 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.335101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tmp-dir\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.335196 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.335166    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tls-certs\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.335345 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.335222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-dshm\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.335345 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.335240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-model-cache\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.436354 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.436327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhgv\" (UniqueName: \"kubernetes.io/projected/f94a4eed-68eb-4de6-b8da-96d486d8db2c-kube-api-access-dfhgv\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.436483 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.436372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-home\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.436483 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.436394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tmp-dir\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.436483 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.436409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tls-certs\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.436483 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.436472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-dshm\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.436703 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.436499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-model-cache\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.436816 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.436790    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-home\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.437007 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.436981    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tmp-dir\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.437210 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.436996    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-model-cache\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.439455 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.439397    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tls-certs\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.439455 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.439401    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-dshm\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.443595 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.443573    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhgv\" (UniqueName: \"kubernetes.io/projected/f94a4eed-68eb-4de6-b8da-96d486d8db2c-kube-api-access-dfhgv\") pod \"e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") " pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.504022 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.503992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:33.779570 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:33.779544    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"]
Sep 17 01:11:33.781778 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:11:33.781755    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf94a4eed_68eb_4de6_b8da_96d486d8db2c.slice/crio-ebd72812c1ff327ec52f643b2dc611db52b59d5a948329d428792423f8676e9f WatchSource:0}: Error finding container ebd72812c1ff327ec52f643b2dc611db52b59d5a948329d428792423f8676e9f: Status 404 returned error can't find the container with id ebd72812c1ff327ec52f643b2dc611db52b59d5a948329d428792423f8676e9f
Sep 17 01:11:34.528701 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:34.528665    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f" event={"ID":"f94a4eed-68eb-4de6-b8da-96d486d8db2c","Type":"ContainerStarted","Data":"3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3"}
Sep 17 01:11:34.528701 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:34.528704    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f" event={"ID":"f94a4eed-68eb-4de6-b8da-96d486d8db2c","Type":"ContainerStarted","Data":"ebd72812c1ff327ec52f643b2dc611db52b59d5a948329d428792423f8676e9f"}
Sep 17 01:11:34.545514 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:34.545460    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f" podStartSLOduration=1.545439192 podStartE2EDuration="1.545439192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:11:34.542897307 +0000 UTC m=+2490.498493661" watchObservedRunningTime="2026-09-17 01:11:34.545439192 +0000 UTC m=+2490.501035534"
Sep 17 01:11:43.504647 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:43.504611    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:43.505117 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:43.504669    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:43.517081 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:43.517061    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:11:43.586444 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:11:43.586416    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:12:05.234976 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:05.234949    2576 scope.go:117] "RemoveContainer" containerID="1ed429e19e7457e71e1ecc448c96559d4b680edf55a43f9b3a9eec95c3e60a00"
Sep 17 01:12:10.159261 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:10.159227    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"]
Sep 17 01:12:10.180126 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:10.180098    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:12:11.321109 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:11.321066    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"]
Sep 17 01:12:11.338240 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:11.338208    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:12:25.179766 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.179717    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4" podUID="40d6e212-b218-4a3b-b472-9e35b034065e" containerName="main" containerID="cri-o://0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8" gracePeriod=45
Sep 17 01:12:25.435350 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.435290    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:12:25.548479 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.548448    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-tmp-dir\") pod \"40d6e212-b218-4a3b-b472-9e35b034065e\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") "
Sep 17 01:12:25.548642 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.548510    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-home\") pod \"40d6e212-b218-4a3b-b472-9e35b034065e\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") "
Sep 17 01:12:25.548642 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.548547    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-dshm\") pod \"40d6e212-b218-4a3b-b472-9e35b034065e\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") "
Sep 17 01:12:25.548642 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.548595    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d6e212-b218-4a3b-b472-9e35b034065e-tls-certs\") pod \"40d6e212-b218-4a3b-b472-9e35b034065e\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") "
Sep 17 01:12:25.548642 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.548621    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-model-cache\") pod \"40d6e212-b218-4a3b-b472-9e35b034065e\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") "
Sep 17 01:12:25.548865 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.548676    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fffx\" (UniqueName: \"kubernetes.io/projected/40d6e212-b218-4a3b-b472-9e35b034065e-kube-api-access-2fffx\") pod \"40d6e212-b218-4a3b-b472-9e35b034065e\" (UID: \"40d6e212-b218-4a3b-b472-9e35b034065e\") "
Sep 17 01:12:25.548865 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.548739    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40d6e212-b218-4a3b-b472-9e35b034065e" (UID: "40d6e212-b218-4a3b-b472-9e35b034065e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:12:25.548865 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.548823    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-home" (OuterVolumeSpecName: "home") pod "40d6e212-b218-4a3b-b472-9e35b034065e" (UID: "40d6e212-b218-4a3b-b472-9e35b034065e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:12:25.548977 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.548904    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-model-cache" (OuterVolumeSpecName: "model-cache") pod "40d6e212-b218-4a3b-b472-9e35b034065e" (UID: "40d6e212-b218-4a3b-b472-9e35b034065e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:12:25.549027 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.549013    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:25.549063 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.549032    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:25.549063 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.549050    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:25.550907 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.550887    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-dshm" (OuterVolumeSpecName: "dshm") pod "40d6e212-b218-4a3b-b472-9e35b034065e" (UID: "40d6e212-b218-4a3b-b472-9e35b034065e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:12:25.551003 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.550966    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d6e212-b218-4a3b-b472-9e35b034065e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40d6e212-b218-4a3b-b472-9e35b034065e" (UID: "40d6e212-b218-4a3b-b472-9e35b034065e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:12:25.551003 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.550995    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d6e212-b218-4a3b-b472-9e35b034065e-kube-api-access-2fffx" (OuterVolumeSpecName: "kube-api-access-2fffx") pod "40d6e212-b218-4a3b-b472-9e35b034065e" (UID: "40d6e212-b218-4a3b-b472-9e35b034065e"). InnerVolumeSpecName "kube-api-access-2fffx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:12:25.649874 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.649851    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d6e212-b218-4a3b-b472-9e35b034065e-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:25.649981 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.649879    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d6e212-b218-4a3b-b472-9e35b034065e-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:25.649981 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.649895    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fffx\" (UniqueName: \"kubernetes.io/projected/40d6e212-b218-4a3b-b472-9e35b034065e-kube-api-access-2fffx\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:25.782122 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.782092    2576 generic.go:356] "Generic (PLEG): container finished" podID="40d6e212-b218-4a3b-b472-9e35b034065e" containerID="0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8" exitCode=0
Sep 17 01:12:25.782260 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.782133    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"
Sep 17 01:12:25.782260 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.782180    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4" event={"ID":"40d6e212-b218-4a3b-b472-9e35b034065e","Type":"ContainerDied","Data":"0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8"}
Sep 17 01:12:25.782260 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.782219    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4" event={"ID":"40d6e212-b218-4a3b-b472-9e35b034065e","Type":"ContainerDied","Data":"ffbcf1c9e7d033fc1ea1dfd79509a9ddb02f5061eaa217570692c1a66118aaa9"}
Sep 17 01:12:25.782260 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.782235    2576 scope.go:117] "RemoveContainer" containerID="0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8"
Sep 17 01:12:25.791556 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.791540    2576 scope.go:117] "RemoveContainer" containerID="0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8"
Sep 17 01:12:25.791821 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:12:25.791803    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8\": container with ID starting with 0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8 not found: ID does not exist" containerID="0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8"
Sep 17 01:12:25.791868 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.791829    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8"} err="failed to get container status \"0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8\": rpc error: code = NotFound desc = could not find container \"0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8\": container with ID starting with 0acaf27995a13fb4894c6aba7a0283259f37346382908c46530a57f45d6ea4d8 not found: ID does not exist"
Sep 17 01:12:25.802277 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.802256    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"]
Sep 17 01:12:25.804234 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:25.804213    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d97c19b1-kserve-68b658cfcf-5ptk4"]
Sep 17 01:12:26.342968 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.342935    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f" podUID="f94a4eed-68eb-4de6-b8da-96d486d8db2c" containerName="main" containerID="cri-o://3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3" gracePeriod=45
Sep 17 01:12:26.591417 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.591262    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d6e212-b218-4a3b-b472-9e35b034065e" path="/var/lib/kubelet/pods/40d6e212-b218-4a3b-b472-9e35b034065e/volumes"
Sep 17 01:12:26.605477 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.605420    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:12:26.659455 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.659426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfhgv\" (UniqueName: \"kubernetes.io/projected/f94a4eed-68eb-4de6-b8da-96d486d8db2c-kube-api-access-dfhgv\") pod \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") "
Sep 17 01:12:26.659583 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.659472    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-model-cache\") pod \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") "
Sep 17 01:12:26.659583 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.659492    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tls-certs\") pod \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") "
Sep 17 01:12:26.659655 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.659601    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-home\") pod \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") "
Sep 17 01:12:26.659690 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.659668    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tmp-dir\") pod \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") "
Sep 17 01:12:26.659731 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.659712    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f94a4eed-68eb-4de6-b8da-96d486d8db2c" (UID: "f94a4eed-68eb-4de6-b8da-96d486d8db2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:12:26.659781 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.659731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-dshm\") pod \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\" (UID: \"f94a4eed-68eb-4de6-b8da-96d486d8db2c\") "
Sep 17 01:12:26.659899 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.659851    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-home" (OuterVolumeSpecName: "home") pod "f94a4eed-68eb-4de6-b8da-96d486d8db2c" (UID: "f94a4eed-68eb-4de6-b8da-96d486d8db2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:12:26.659971 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.659893    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f94a4eed-68eb-4de6-b8da-96d486d8db2c" (UID: "f94a4eed-68eb-4de6-b8da-96d486d8db2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:12:26.660369 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.660064    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-model-cache\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:26.660369 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.660325    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-home\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:26.660369 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.660340    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tmp-dir\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:26.661775 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.661747    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94a4eed-68eb-4de6-b8da-96d486d8db2c-kube-api-access-dfhgv" (OuterVolumeSpecName: "kube-api-access-dfhgv") pod "f94a4eed-68eb-4de6-b8da-96d486d8db2c" (UID: "f94a4eed-68eb-4de6-b8da-96d486d8db2c"). InnerVolumeSpecName "kube-api-access-dfhgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 01:12:26.661936 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.661909    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-dshm" (OuterVolumeSpecName: "dshm") pod "f94a4eed-68eb-4de6-b8da-96d486d8db2c" (UID: "f94a4eed-68eb-4de6-b8da-96d486d8db2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 01:12:26.662033 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.661938    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f94a4eed-68eb-4de6-b8da-96d486d8db2c" (UID: "f94a4eed-68eb-4de6-b8da-96d486d8db2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 01:12:26.760932 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.760910    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a4eed-68eb-4de6-b8da-96d486d8db2c-dshm\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:26.760932 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.760930    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfhgv\" (UniqueName: \"kubernetes.io/projected/f94a4eed-68eb-4de6-b8da-96d486d8db2c-kube-api-access-dfhgv\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:26.761062 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.760939    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a4eed-68eb-4de6-b8da-96d486d8db2c-tls-certs\") on node \"ip-10-0-143-51.ec2.internal\" DevicePath \"\""
Sep 17 01:12:26.789208 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.789172    2576 generic.go:356] "Generic (PLEG): container finished" podID="f94a4eed-68eb-4de6-b8da-96d486d8db2c" containerID="3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3" exitCode=0
Sep 17 01:12:26.789298 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.789208    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"
Sep 17 01:12:26.789298 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.789228    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f" event={"ID":"f94a4eed-68eb-4de6-b8da-96d486d8db2c","Type":"ContainerDied","Data":"3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3"}
Sep 17 01:12:26.789298 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.789268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f" event={"ID":"f94a4eed-68eb-4de6-b8da-96d486d8db2c","Type":"ContainerDied","Data":"ebd72812c1ff327ec52f643b2dc611db52b59d5a948329d428792423f8676e9f"}
Sep 17 01:12:26.789298 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.789286    2576 scope.go:117] "RemoveContainer" containerID="3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3"
Sep 17 01:12:26.797821 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.797807    2576 scope.go:117] "RemoveContainer" containerID="3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3"
Sep 17 01:12:26.798052 ip-10-0-143-51 kubenswrapper[2576]: E0917 01:12:26.798036    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3\": container with ID starting with 3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3 not found: ID does not exist" containerID="3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3"
Sep 17 01:12:26.798123 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.798059    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3"} err="failed to get container status \"3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3\": rpc error: code = NotFound desc = could not find container \"3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3\": container with ID starting with 3badbab58957a0e8a0d3d1d02cce89de9f68ec1939f23c3e7cd41009e2d705f3 not found: ID does not exist"
Sep 17 01:12:26.808595 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.808572    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"]
Sep 17 01:12:26.810338 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:26.810320    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d97c19b1-kserve-675db8ff55-q5c6f"]
Sep 17 01:12:28.588297 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:28.588264    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f94a4eed-68eb-4de6-b8da-96d486d8db2c" path="/var/lib/kubelet/pods/f94a4eed-68eb-4de6-b8da-96d486d8db2c/volumes"
Sep 17 01:12:36.286997 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:12:36.286963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:13:05.254525 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:05.254495    2576 scope.go:117] "RemoveContainer" containerID="b3f6f5a83b4ffd4bed2dbb3b4d8411cba43487bd87c14e1d8ba418ccf18fde2b"
Sep 17 01:13:05.262616 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:05.262597    2576 scope.go:117] "RemoveContainer" containerID="47db192626db65bd05c05ccbcc74e5fe6fe1c97d68027d379a422fcfd2d876d3"
Sep 17 01:13:05.269919 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:05.269903    2576 scope.go:117] "RemoveContainer" containerID="863661966d87aeb54987f4ecb0f201a8bd322d4194fa2ea698098cda891c3923"
Sep 17 01:13:06.990701 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:06.990661    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ccv7w_490c1f14-ec34-405a-aa6f-aef946f1d204/manager/0.log"
Sep 17 01:13:07.113485 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:07.113458    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kzgnq_2fd22045-cb5b-4fd1-bcea-47e592e0abe1/manager/0.log"
Sep 17 01:13:07.357166 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:07.357109    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hk8l5_35441bf9-fae9-40d9-90a4-443a82ef0921/server/0.log"
Sep 17 01:13:07.481884 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:07.481857    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h5kxq_009e1fc3-a69e-47c3-924a-ab5adc7066f5/manager/0.log"
Sep 17 01:13:07.733915 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:07.733830    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596c59f645-k4b9n_b3b2214f-12a7-4d14-a367-0f6410d9a4fc/manager/0.log"
Sep 17 01:13:07.739954 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:07.739928    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596c59f645-k4b9n_b3b2214f-12a7-4d14-a367-0f6410d9a4fc/copy-manifests/0.log"
Sep 17 01:13:08.436047 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:08.436023    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826765-5v5nm_bbb52972-2499-4419-8d3f-e02751747455/cleanup/0.log"
Sep 17 01:13:08.545364 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:08.545342    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826780-hwvzb_a98d8393-d4db-4241-ac6c-b9c32e3724a3/cleanup/0.log"
Sep 17 01:13:08.653115 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:08.653089    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-n7q4c_5eea5411-2230-40cc-afa4-b672b56ec1f0/postgres/0.log"
Sep 17 01:13:09.873804 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:09.873770    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hbks6_95cdd121-9a20-49ab-a6c9-c9f1967064f1/manager/0.log"
Sep 17 01:13:10.931557 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:10.931526    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-dgjf4_7d869572-a142-4660-bf5c-d5f17d425672/istio-proxy/1.log"
Sep 17 01:13:11.050782 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:11.050757    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-jfwzp_970c5f58-40ca-44c5-9c57-86ba92cc9ec1/payload-pre-processing/0.log"
Sep 17 01:13:11.283753 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:11.283726    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6947d5777d-d7tnw_78ce94d3-caf0-42e3-85a0-728042777ffa/router/0.log"
Sep 17 01:13:11.630359 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:11.630286    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5nxkg_c0bea1a0-f99d-46fd-b850-211bdfb4d794/main/0.log"
Sep 17 01:13:11.861691 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:11.861666    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-4c2n7_a0b28f85-aef9-47ea-8236-900bd042c1f1/main/0.log"
Sep 17 01:13:17.630912 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.630867    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jsqsk/must-gather-n68cv"]
Sep 17 01:13:17.631473 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.631456    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d6e212-b218-4a3b-b472-9e35b034065e" containerName="main"
Sep 17 01:13:17.631519 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.631477    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d6e212-b218-4a3b-b472-9e35b034065e" containerName="main"
Sep 17 01:13:17.631519 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.631497    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94a4eed-68eb-4de6-b8da-96d486d8db2c" containerName="main"
Sep 17 01:13:17.631519 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.631505    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94a4eed-68eb-4de6-b8da-96d486d8db2c" containerName="main"
Sep 17 01:13:17.631612 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.631590    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f94a4eed-68eb-4de6-b8da-96d486d8db2c" containerName="main"
Sep 17 01:13:17.631612 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.631602    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d6e212-b218-4a3b-b472-9e35b034065e" containerName="main"
Sep 17 01:13:17.635072 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.635054    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jsqsk/must-gather-n68cv"
Sep 17 01:13:17.637392 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.637366    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jsqsk\"/\"openshift-service-ca.crt\""
Sep 17 01:13:17.637528 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.637403    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jsqsk\"/\"kube-root-ca.crt\""
Sep 17 01:13:17.637528 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.637509    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jsqsk\"/\"default-dockercfg-bt7hf\""
Sep 17 01:13:17.644191 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.644165    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jsqsk/must-gather-n68cv"]
Sep 17 01:13:17.645826 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.645801    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:13:17.707684 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.707651    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:13:17.763701 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.763674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6tj\" (UniqueName: \"kubernetes.io/projected/a87a0984-df6b-4aaf-ae63-98e5a21199f3-kube-api-access-kg6tj\") pod \"must-gather-n68cv\" (UID: \"a87a0984-df6b-4aaf-ae63-98e5a21199f3\") " pod="openshift-must-gather-jsqsk/must-gather-n68cv"
Sep 17 01:13:17.763843 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.763753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a87a0984-df6b-4aaf-ae63-98e5a21199f3-must-gather-output\") pod \"must-gather-n68cv\" (UID: \"a87a0984-df6b-4aaf-ae63-98e5a21199f3\") " pod="openshift-must-gather-jsqsk/must-gather-n68cv"
Sep 17 01:13:17.864948 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.864922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6tj\" (UniqueName: \"kubernetes.io/projected/a87a0984-df6b-4aaf-ae63-98e5a21199f3-kube-api-access-kg6tj\") pod \"must-gather-n68cv\" (UID: \"a87a0984-df6b-4aaf-ae63-98e5a21199f3\") " pod="openshift-must-gather-jsqsk/must-gather-n68cv"
Sep 17 01:13:17.865063 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.865023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a87a0984-df6b-4aaf-ae63-98e5a21199f3-must-gather-output\") pod \"must-gather-n68cv\" (UID: \"a87a0984-df6b-4aaf-ae63-98e5a21199f3\") " pod="openshift-must-gather-jsqsk/must-gather-n68cv"
Sep 17 01:13:17.865370 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.865354    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a87a0984-df6b-4aaf-ae63-98e5a21199f3-must-gather-output\") pod \"must-gather-n68cv\" (UID: \"a87a0984-df6b-4aaf-ae63-98e5a21199f3\") " pod="openshift-must-gather-jsqsk/must-gather-n68cv"
Sep 17 01:13:17.872594 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.872574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6tj\" (UniqueName: \"kubernetes.io/projected/a87a0984-df6b-4aaf-ae63-98e5a21199f3-kube-api-access-kg6tj\") pod \"must-gather-n68cv\" (UID: \"a87a0984-df6b-4aaf-ae63-98e5a21199f3\") " pod="openshift-must-gather-jsqsk/must-gather-n68cv"
Sep 17 01:13:17.945584 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:17.945532    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jsqsk/must-gather-n68cv"
Sep 17 01:13:18.222058 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:18.221989    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jsqsk/must-gather-n68cv"]
Sep 17 01:13:18.225727 ip-10-0-143-51 kubenswrapper[2576]: W0917 01:13:18.225700    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda87a0984_df6b_4aaf_ae63_98e5a21199f3.slice/crio-23a38c3d6623624aeb5e2e5880576ebe582ceca5574f368c1030f4de795e6cb4 WatchSource:0}: Error finding container 23a38c3d6623624aeb5e2e5880576ebe582ceca5574f368c1030f4de795e6cb4: Status 404 returned error can't find the container with id 23a38c3d6623624aeb5e2e5880576ebe582ceca5574f368c1030f4de795e6cb4
Sep 17 01:13:18.227726 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:18.227708    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 01:13:19.052797 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:19.052770    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jsqsk/must-gather-n68cv" event={"ID":"a87a0984-df6b-4aaf-ae63-98e5a21199f3","Type":"ContainerStarted","Data":"23a38c3d6623624aeb5e2e5880576ebe582ceca5574f368c1030f4de795e6cb4"}
Sep 17 01:13:20.061709 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:20.061671    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jsqsk/must-gather-n68cv" event={"ID":"a87a0984-df6b-4aaf-ae63-98e5a21199f3","Type":"ContainerStarted","Data":"695499b8d3923a88a925666559b96450275beb60e9ff525a8eefbf6a418c1475"}
Sep 17 01:13:20.062209 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:20.062083    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jsqsk/must-gather-n68cv" event={"ID":"a87a0984-df6b-4aaf-ae63-98e5a21199f3","Type":"ContainerStarted","Data":"ef1da259c9f8463f9e83b41cd9219633a6eb6e15eef710934335944a21fc43cf"}
Sep 17 01:13:20.079701 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:20.079648    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jsqsk/must-gather-n68cv" podStartSLOduration=2.221788799 podStartE2EDuration="3.079632921s" totalImagesPullingTime="857.844122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:13:20.078104094 +0000 UTC m=+2596.033700435" watchObservedRunningTime="2026-09-17 01:13:20.079632921 +0000 UTC m=+2596.035229262"
Sep 17 01:13:20.613806 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:20.613775    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mlq9c_289b9bc3-1131-473a-a889-0cdb22286663/global-pull-secret-syncer/0.log"
Sep 17 01:13:20.664877 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:20.664842    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log"
Sep 17 01:13:20.787017 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:20.786989    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-51.ec2.internal_b66c31ea21941fb1ad957ccbcb0a0288/haproxy/0.log"
Sep 17 01:13:26.055655 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:26.055623    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hbks6_95cdd121-9a20-49ab-a6c9-c9f1967064f1/manager/0.log"
Sep 17 01:13:27.551580 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:27.551444    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-8dbdr_77a48136-1b49-4db8-8b78-a3270c01f775/cluster-monitoring-operator/0.log"
Sep 17 01:13:27.672011 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:27.671967    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-7htk7_ebad7518-ce1b-4ea3-8d53-82acba867e65/monitoring-plugin/0.log"
Sep 17 01:13:27.701395 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:27.701354    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gbhrp_502e8041-e346-4b4c-a6ee-8d39143b4003/node-exporter/0.log"
Sep 17 01:13:27.722903 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:27.722872    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gbhrp_502e8041-e346-4b4c-a6ee-8d39143b4003/kube-rbac-proxy/0.log"
Sep 17 01:13:27.742394 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:27.742360    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gbhrp_502e8041-e346-4b4c-a6ee-8d39143b4003/init-textfile/0.log"
Sep 17 01:13:28.051476 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:28.051395    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e787bd95-8de8-4109-b922-f0730905c907/prometheus/0.log"
Sep 17 01:13:28.079915 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:28.079883    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e787bd95-8de8-4109-b922-f0730905c907/config-reloader/0.log"
Sep 17 01:13:28.102938 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:28.102906    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e787bd95-8de8-4109-b922-f0730905c907/thanos-sidecar/0.log"
Sep 17 01:13:28.122268 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:28.122180    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e787bd95-8de8-4109-b922-f0730905c907/kube-rbac-proxy-web/0.log"
Sep 17 01:13:28.142743 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:28.142714    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e787bd95-8de8-4109-b922-f0730905c907/kube-rbac-proxy/0.log"
Sep 17 01:13:28.162543 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:28.162519    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e787bd95-8de8-4109-b922-f0730905c907/kube-rbac-proxy-thanos/0.log"
Sep 17 01:13:28.182965 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:28.181709    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e787bd95-8de8-4109-b922-f0730905c907/init-config-reloader/0.log"
Sep 17 01:13:30.345655 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.345614    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"]
Sep 17 01:13:30.349917 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.349892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.358922 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.358894    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"]
Sep 17 01:13:30.400507 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.400468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-lib-modules\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.400704 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.400531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-proc\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.400704 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.400590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-sys\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.400704 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.400656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-podres\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.400884 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.400704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvfbd\" (UniqueName: \"kubernetes.io/projected/55ee430a-53fb-4475-a8fb-60a5d9c9e480-kube-api-access-gvfbd\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.502190 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.502127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-lib-modules\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.502384 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.502224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-proc\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.502384 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.502257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-sys\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.502384 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.502357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-sys\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.502562 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.502427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-podres\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.502562 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.502498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvfbd\" (UniqueName: \"kubernetes.io/projected/55ee430a-53fb-4475-a8fb-60a5d9c9e480-kube-api-access-gvfbd\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.502989 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.502958    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-lib-modules\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.503100 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.503027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-proc\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.503190 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.503112    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55ee430a-53fb-4475-a8fb-60a5d9c9e480-podres\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.520641 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.520608    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvfbd\" (UniqueName: \"kubernetes.io/projected/55ee430a-53fb-4475-a8fb-60a5d9c9e480-kube-api-access-gvfbd\") pod \"perf-node-gather-daemonset-kltmv\" (UID: \"55ee430a-53fb-4475-a8fb-60a5d9c9e480\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.662508 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.662426    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:30.965823 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:30.965790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"]
Sep 17 01:13:31.142309 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:31.142268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv" event={"ID":"55ee430a-53fb-4475-a8fb-60a5d9c9e480","Type":"ContainerStarted","Data":"366fa8b75120bd7d76f288324ba0e08c010f4d30bdd7241c130a14cdc8828e4a"}
Sep 17 01:13:31.142508 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:31.142335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv" event={"ID":"55ee430a-53fb-4475-a8fb-60a5d9c9e480","Type":"ContainerStarted","Data":"12461bc1519341ef347afe7bceb52c8c8959e48605b6d71fca990f1b85dd5b98"}
Sep 17 01:13:31.143127 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:31.143098    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:31.158268 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:31.158224    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv" podStartSLOduration=1.158205336 podStartE2EDuration="1.158205336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:13:31.157610996 +0000 UTC m=+2607.113207337" watchObservedRunningTime="2026-09-17 01:13:31.158205336 +0000 UTC m=+2607.113801688"
Sep 17 01:13:31.580349 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:31.580316    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-zpnc4_de73b2a9-de5e-4be3-a40e-4a2bb8c943c7/volume-data-source-validator/0.log"
Sep 17 01:13:32.388528 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:32.388491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wr6h_59884e4c-8c0b-4b73-9c4a-b9d0490ad534/konnectivity-agent/0.log" limit=1024
Sep 17 01:13:32.766121 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:32.766008    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r4k48_bc042542-1f09-45d0-b848-cbf7f11ff4e7/dns/0.log"
Sep 17 01:13:32.785326 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:32.785293    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r4k48_bc042542-1f09-45d0-b848-cbf7f11ff4e7/kube-rbac-proxy/0.log"
Sep 17 01:13:32.891944 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:32.891908    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vhmzk_c99ec850-d4b6-4541-90a4-d158112c2570/dns-node-resolver/0.log"
Sep 17 01:13:33.400030 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:33.400000    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cf4c4_42d25648-164b-4ad8-9d6e-d38f1f3301e4/node-ca/0.log"
Sep 17 01:13:34.741549 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:34.741512    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-dgjf4_7d869572-a142-4660-bf5c-d5f17d425672/istio-proxy/0.log"
Sep 17 01:13:34.747327 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:34.747303    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-dgjf4_7d869572-a142-4660-bf5c-d5f17d425672/istio-proxy/1.log"
Sep 17 01:13:34.785153 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:34.785127    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-jfwzp_970c5f58-40ca-44c5-9c57-86ba92cc9ec1/payload-pre-processing/0.log"
Sep 17 01:13:34.840679 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:34.840649    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6947d5777d-d7tnw_78ce94d3-caf0-42e3-85a0-728042777ffa/router/0.log"
Sep 17 01:13:35.724732 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:35.724706    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x6lxh_f5c1d968-4bc7-4e1b-acea-1388a6a0ffdc/serve-healthcheck-canary/0.log"
Sep 17 01:13:36.302345 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:36.302316    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vsc4b_ced1b4b1-4a7b-4365-8927-bffa7b60740d/kube-rbac-proxy/0.log"
Sep 17 01:13:36.323354 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:36.323319    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vsc4b_ced1b4b1-4a7b-4365-8927-bffa7b60740d/exporter/0.log"
Sep 17 01:13:36.342657 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:36.342630    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vsc4b_ced1b4b1-4a7b-4365-8927-bffa7b60740d/extractor/0.log"
Sep 17 01:13:38.165538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:38.165501    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-kltmv"
Sep 17 01:13:38.383006 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:38.382972    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ccv7w_490c1f14-ec34-405a-aa6f-aef946f1d204/manager/0.log"
Sep 17 01:13:38.774734 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:38.774704    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kzgnq_2fd22045-cb5b-4fd1-bcea-47e592e0abe1/manager/0.log"
Sep 17 01:13:38.851873 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:38.851819    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hk8l5_35441bf9-fae9-40d9-90a4-443a82ef0921/server/0.log"
Sep 17 01:13:38.983431 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:38.983395    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h5kxq_009e1fc3-a69e-47c3-924a-ab5adc7066f5/manager/0.log"
Sep 17 01:13:39.097993 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:39.097961    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596c59f645-k4b9n_b3b2214f-12a7-4d14-a367-0f6410d9a4fc/manager/0.log"
Sep 17 01:13:39.116245 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:39.116218    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596c59f645-k4b9n_b3b2214f-12a7-4d14-a367-0f6410d9a4fc/copy-manifests/0.log"
Sep 17 01:13:40.564982 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:40.564899    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68945d9444-5cvgh_f8be64ce-0dcc-4d26-9113-5efa0ef62a7b/manager/0.log"
Sep 17 01:13:40.609862 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:40.609837    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qjw2x_ebbc5c7a-af9b-4d4d-b17f-7cae1718d303/openshift-lws-operator/0.log"
Sep 17 01:13:41.508993 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:41.508967    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-dgjf4_7d869572-a142-4660-bf5c-d5f17d425672/istio-proxy/0.log"
Sep 17 01:13:41.513462 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:41.513437    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-dgjf4_7d869572-a142-4660-bf5c-d5f17d425672/istio-proxy/1.log"
Sep 17 01:13:41.539962 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:41.539933    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-jfwzp_970c5f58-40ca-44c5-9c57-86ba92cc9ec1/payload-pre-processing/0.log"
Sep 17 01:13:41.578318 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:41.578286    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6947d5777d-d7tnw_78ce94d3-caf0-42e3-85a0-728042777ffa/router/0.log"
Sep 17 01:13:43.454298 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:43.454062    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-dgjf4_7d869572-a142-4660-bf5c-d5f17d425672/istio-proxy/0.log"
Sep 17 01:13:43.459985 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:43.459952    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-dgjf4_7d869572-a142-4660-bf5c-d5f17d425672/istio-proxy/1.log"
Sep 17 01:13:43.484029 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:43.483995    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-jfwzp_970c5f58-40ca-44c5-9c57-86ba92cc9ec1/payload-pre-processing/0.log"
Sep 17 01:13:43.525133 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:43.525109    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6947d5777d-d7tnw_78ce94d3-caf0-42e3-85a0-728042777ffa/router/0.log"
Sep 17 01:13:47.057133 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.057109    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84cm9_50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8/kube-multus-additional-cni-plugins/0.log"
Sep 17 01:13:47.078906 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.078882    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84cm9_50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8/egress-router-binary-copy/0.log"
Sep 17 01:13:47.098315 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.098294    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84cm9_50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8/cni-plugins/0.log"
Sep 17 01:13:47.116252 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.116220    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84cm9_50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8/bond-cni-plugin/0.log"
Sep 17 01:13:47.143538 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.143521    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84cm9_50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8/routeoverride-cni/0.log"
Sep 17 01:13:47.161272 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.161250    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84cm9_50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8/whereabouts-cni-bincopy/0.log"
Sep 17 01:13:47.178766 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.178745    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84cm9_50c88f96-54de-4bbc-a2f1-7b0fb6ff18e8/whereabouts-cni/0.log"
Sep 17 01:13:47.287588 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.287562    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rdtmd_28bc9500-7f38-4e77-98c7-86001cac7fd9/kube-multus/0.log"
Sep 17 01:13:47.431057 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.430975    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w9nj2_06683d51-91e1-4e99-a67d-75d8a715b5ab/network-metrics-daemon/0.log"
Sep 17 01:13:47.447659 ip-10-0-143-51 kubenswrapper[2576]: I0917 01:13:47.447609    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w9nj2_06683d51-91e1-4e99-a67d-75d8a715b5ab/kube-rbac-proxy/0.log"