Jul 24 17:03:16.062922 ip-10-0-140-80 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 17:03:16.411269 ip-10-0-140-80 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:03:16.411269 ip-10-0-140-80 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 17:03:16.411269 ip-10-0-140-80 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:03:16.411269 ip-10-0-140-80 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 17:03:16.411269 ip-10-0-140-80 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:03:16.412918 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.412841    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 17:03:16.416292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416278    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:03:16.416292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416292    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416297    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416300    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416303    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416306    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416308    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416311    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416314    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416316    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416319    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416321    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416324    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416326    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416329    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416332    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416340    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416343    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416346    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416348    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416351    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:03:16.416361 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416353    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416355    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416358    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416360    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416363    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416366    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416369    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416372    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416374    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416377    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416379    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416381    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416385    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416389    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416392    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416395    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416398    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416400    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416403    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:03:16.416847 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416405    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416407    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416411    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416413    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416416    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416418    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416420    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416422    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416425    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416427    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416430    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416432    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416435    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416440    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416443    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416446    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416449    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416452    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416454    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:03:16.417292 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416457    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416459    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416462    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416464    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416467    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416469    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416472    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416474    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416476    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416479    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416481    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416484    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416486    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416488    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416491    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416493    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416496    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416499    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416501    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416504    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:03:16.417835 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416506    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416508    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416512    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416514    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416517    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416519    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416522    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416525    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416528    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416531    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416533    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416536    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416538    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416541    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416543    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416546    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416548    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416552    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416555    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:03:16.418329 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.416558    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417235    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417242    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417245    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417248    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417252    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417255    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417257    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417260    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417262    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417265    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417268    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417270    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417272    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417275    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417278    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417281    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417283    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417286    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417289    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:03:16.418809 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417292    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417296    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417300    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417304    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417308    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417311    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417313    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417316    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417319    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417321    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417324    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417326    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417329    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417331    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417334    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417336    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417339    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417341    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417344    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:03:16.419299 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417346    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417349    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417351    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417353    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417356    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417359    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417361    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417364    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417366    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417368    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417371    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417373    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417376    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417378    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417381    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417383    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417386    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417388    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417391    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:03:16.419804 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417394    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417396    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417398    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417401    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417403    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417406    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417409    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417411    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417414    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417416    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417420    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417422    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417425    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417427    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417429    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417432    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417434    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417436    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417439    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417442    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:03:16.420281 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417445    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417448    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417451    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417453    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417456    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417458    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417461    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417464    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417466    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417469    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417471    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417474    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417476    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417478    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417481    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417483    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417485    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417488    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417492    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:03:16.420785 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417495    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417498    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417501    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417864    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417869    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417872    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417875    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417878    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417881    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417884    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417887    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417890    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417892    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417895    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417897    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417900    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417902    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417905    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417907    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417911    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:03:16.421237 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417915    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417918    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417920    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417923    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417925    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417928    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417930    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417933    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417935    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417938    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417940    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417943    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417946    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417948    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417951    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417953    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417955    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417958    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417960    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417962    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:03:16.421770 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417965    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417968    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417971    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417973    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417976    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417978    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417980    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417983    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417986    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417988    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417991    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417993    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417996    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.417999    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418001    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418004    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418006    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418008    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418011    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418013    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:03:16.422239 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418016    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418020    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418022    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418025    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418028    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418031    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418034    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418036    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418039    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418041    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418044    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418047    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418049    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418052    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418054    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418057    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418059    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418062    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418064    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:03:16.422796 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418068    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418070    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418072    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418077    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418080    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418083    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418086    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418088    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418091    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418093    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418099    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418101    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418104    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418106    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418108    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418111    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418113    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418116    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418118    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418121    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:03:16.423320 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418123    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418127    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418130    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418195    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418202    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418208    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418212    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418216    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418219    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418223    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418226    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418230    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418233    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418236    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418239    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418243    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418246    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418249    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418252    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418256    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418258    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418262    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418265    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 17:03:16.423823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418269    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418272    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418276    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418279    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418282    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418285    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418288    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418291    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418294    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418296    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418299    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418302    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418306    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418309    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418311    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418314    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418317    2563 flags.go:64] FLAG: --enable-server="true"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418320    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418325    2563 flags.go:64] FLAG: --event-burst="100"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418328    2563 flags.go:64] FLAG: --event-qps="50"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418331    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418335    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418338    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418341    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 17:03:16.424411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418346    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418349    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418352    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418355    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418358    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418360    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418363    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418366    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418369    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418372    2563 flags.go:64] FLAG: --feature-gates=""
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418375    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418378    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418381    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418383    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418386    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418389    2563 flags.go:64] FLAG: --help="false"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418392    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418395    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418397    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418400    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418403    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418406    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418409    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 17:03:16.425041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418411    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418414    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418417    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418419    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418422    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418425    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418428    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418431    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418434    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418439    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418442    2563 flags.go:64] FLAG: --lock-file=""
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418445    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418448    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418451    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418456    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418459    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418461    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418464    2563 flags.go:64] FLAG: --logging-format="text"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418467    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418470    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418473    2563 flags.go:64] FLAG: --manifest-url=""
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418476    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418480    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418482    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418486    2563 flags.go:64] FLAG: --max-pods="110"
Jul 24 17:03:16.425629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418489    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418492    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418495    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418497    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418500    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418503    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418505    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418511    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418514    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418517    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418520    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418523    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418527    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418530    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418533    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418536    2563 flags.go:64] FLAG: --port="10250"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418539    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418542    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a406a07cffeb95f4"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418547    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418550    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418553    2563 flags.go:64] FLAG: --register-node="true"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418555    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418559    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418562    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 24 17:03:16.426232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418564    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418567    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418570    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418573    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418576    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418578    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418581    2563 flags.go:64] FLAG: --runonce="false"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418584    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418586    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418589    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418592    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418595    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418597    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418600    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418603    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418605    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418608    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418611    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418613    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418616    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418619    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418621    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418627    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418629    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418632    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 17:03:16.426825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418637    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418641    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418657    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418660    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418663    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418666    2563 flags.go:64] FLAG: --v="2"
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418670    2563 flags.go:64] FLAG: --version="false"
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418673    2563 flags.go:64] FLAG: --vmodule=""
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418677    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.418680    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418784    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418788    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418791    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418793    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418796    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418798    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418801    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418803    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418806    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418808    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418811    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418813    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:03:16.427430 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418816    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418818    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418821    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418823    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418826    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418828    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418831    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418833    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418835    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418838    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418843    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418847    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418849    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418851    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418854    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418856    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418859    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418861    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418864    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:03:16.428017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418866    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418868    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418871    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418874    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418876    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418879    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418881    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418885    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418888    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418891    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418895    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418898    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418900    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418903    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418906    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418909    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418911    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418914    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418916    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:03:16.428487 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418919    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418921    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418923    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418926    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418930    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418934    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418937    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418939    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418942    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418945    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418949    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418952    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418954    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418957    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418959    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418962    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418964    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418967    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418969    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418971    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:03:16.429031 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418974    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418976    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418979    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418981    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418983    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418986    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418988    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418991    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418993    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418996    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.418998    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419001    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419003    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419005    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419008    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419010    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419015    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419018    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419021    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419024    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:03:16.429514 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419026    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419029    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419031    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419033    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419036    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419038    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419041    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419043    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.419046    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.419600    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.426332    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.426345    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.427044    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.427051    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427114    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:03:16.430017 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427120    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427123    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427126    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427129    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427138    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427140    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427143    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427145    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427148    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427151    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427153    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427156    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427158    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427161    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427163    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427171    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427174    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427176    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427178    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427181    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:03:16.430387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427183    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427186    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427188    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427191    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427193    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427195    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427198    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427202    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427206    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427209    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427212    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427214    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427217    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427220    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427223    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427226    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427228    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427231    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427234    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:03:16.430897 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427236    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427239    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427242    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427244    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427247    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427250    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427253    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427255    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427257    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427268    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427272    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427274    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427277    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427279    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427282    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427285    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427287    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427290    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427292    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427295    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:03:16.431396 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427297    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427300    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427303    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427305    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427308    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427310    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427312    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427315    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427318    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427320    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427323    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427325    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427328    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427330    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427332    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427335    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427337    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427339    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427342    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427344    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:03:16.431900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427346    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427349    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427360    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427362    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427365    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427367    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427369    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427372    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427374    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427378    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427381    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427384    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427387    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427389    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427392    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427394    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427397    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427399    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:03:16.432377 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427401    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.427406    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427543    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427547    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427550    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427553    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427555    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427559    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427563    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427566    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427569    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427572    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427574    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427577    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:03:16.432810 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427579    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427581    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427584    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427594    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427597    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427599    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427601    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427604    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427606    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427610    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427613    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427616    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427618    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427620    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427623    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427625    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427628    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427630    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427632    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:03:16.433153 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427635    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427637    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427639    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427642    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427656    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427659    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427662    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427664    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427666    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427669    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427671    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427674    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427676    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427678    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427681    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427683    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427686    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427694    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427696    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427699    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:03:16.433597 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427701    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427704    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427706    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427709    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427711    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427713    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427716    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427720    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427723    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427726    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427729    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427732    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427734    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427737    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427739    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427742    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427744    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427746    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427749    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:03:16.434082 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427751    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427754    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427756    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427758    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427761    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427763    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427765    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427768    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427770    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427773    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427775    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427783    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427786    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427788    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427791    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427793    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427796    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427798    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427800    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427803    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:03:16.434528 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427805    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427808    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427810    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427813    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427815    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427818    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427820    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427822    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:16.427825    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.427830    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.427932    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.431445    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.432729    2563 server.go:1019] "Starting client certificate rotation"
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.432843    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 17:03:16.435024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.433546    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 17:03:16.435386 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.435068    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 17:03:16.440141 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.440120    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 17:03:16.449947 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.449924    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 24 17:03:16.451995 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.451981    2563 log.go:25] "Validated CRI v1 image API"
Jul 24 17:03:16.453508 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.453490    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 17:03:16.456860 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.456843    2563 fs.go:135] Filesystem UUIDs: map[720a98e1-f449-47b1-ab85-b242e0611307:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d7823c34-1d1a-4741-afe5-3d61565fab62:/dev/nvme0n1p3]
Jul 24 17:03:16.456935 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.456859    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 17:03:16.460939 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.460841    2563 manager.go:216] Machine: {Timestamp:2026-07-24 17:03:16.460358597 +0000 UTC m=+0.291611277 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104294 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a0fc2e4a56c6d3cb4eed7e901610 SystemUUID:ec24a0fc-2e4a-56c6-d3cb-4eed7e901610 BootID:b3a209a8-8b2c-4a2a-9baa-f3c49344d157 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:eb:58:c6:ac:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:eb:58:c6:ac:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:37:53:9b:1a:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 17:03:16.460939 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.460933    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 17:03:16.461036 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.461000    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 17:03:16.462086 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.462064    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 17:03:16.462216 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.462085    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 17:03:16.462776 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.462767    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 17:03:16.462806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.462778    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 17:03:16.462806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.462785    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 17:03:16.462806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.462795    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 17:03:16.463738 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.463728    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 17:03:16.466405 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.466393    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 17:03:16.466527 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.466518    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 17:03:16.466564 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.466549    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 17:03:16.466592 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.466565    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 17:03:16.466619 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.466594    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 17:03:16.466619 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.466604    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 24 17:03:16.466619 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.466616    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 17:03:16.467178 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.467165    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 17:03:16.467223 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.467187    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 17:03:16.467678 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.467658    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 17:03:16.469211 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.469180    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 17:03:16.473530 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.473498    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 17:03:16.473592 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.473554    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 17:03:16.475337 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475322    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 17:03:16.475337 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475341    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 17:03:16.475337 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475346    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 17:03:16.475337 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475352    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 17:03:16.475499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475360    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 17:03:16.475499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475365    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 17:03:16.475499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475373    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 17:03:16.475499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475378    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 17:03:16.475499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475384    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 17:03:16.475499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475389    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 17:03:16.475499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475402    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 17:03:16.475499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.475407    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 17:03:16.476159 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.476148    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 17:03:16.476159 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.476158    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 17:03:16.476367 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.476347    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 17:03:16.476367 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.476346    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 17:03:16.478126 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.478109    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 17:03:16.479218 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.479203    2563 server.go:1268] "Started kubelet"
Jul 24 17:03:16.479317 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.479289    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 17:03:16.479317 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.479284    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 17:03:16.479404 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.479365    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 17:03:16.479875 ip-10-0-140-80 systemd[1]: Started Kubernetes Kubelet.
Jul 24 17:03:16.481520 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.481501    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 17:03:16.481607 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.481571    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 17:03:16.484627 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.484608    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 17:03:16.487190 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.486406    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-80.ec2.internal.18c5488cb442e6ff  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-80.ec2.internal,UID:ip-10-0-140-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-80.ec2.internal,},FirstTimestamp:2026-07-24 17:03:16.479174399 +0000 UTC m=+0.310427077,LastTimestamp:2026-07-24 17:03:16.479174399 +0000 UTC m=+0.310427077,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-80.ec2.internal,}"
Jul 24 17:03:16.488901 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.488883    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 17:03:16.488956 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.488887    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 17:03:16.489008 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.488999    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:16.489538 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.489518    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 17:03:16.489538 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.489521    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 17:03:16.489705 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.489549    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 17:03:16.489705 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.489684    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 17:03:16.489705 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.489693    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 17:03:16.489975 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.489934    2563 factory.go:55] Registering systemd factory
Jul 24 17:03:16.489975 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.489948    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 24 17:03:16.490160 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.490140    2563 factory.go:153] Registering CRI-O factory
Jul 24 17:03:16.490160 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.490154    2563 factory.go:223] Registration of the crio container factory successfully
Jul 24 17:03:16.490267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.490205    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 17:03:16.490267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.490229    2563 factory.go:103] Registering Raw factory
Jul 24 17:03:16.490267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.490244    2563 manager.go:1196] Started watching for new ooms in manager
Jul 24 17:03:16.490734 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.490720    2563 manager.go:349] Starting recovery of all containers
Jul 24 17:03:16.491228 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.491169    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 17:03:16.491359 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.491332    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 17:03:16.492372 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.492350    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nvlsv"
Jul 24 17:03:16.500370 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.500226    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nvlsv"
Jul 24 17:03:16.501901 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.501871    2563 manager.go:354] Recovery completed
Jul 24 17:03:16.506511 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.506497    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:03:16.508477 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.508463    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:03:16.508543 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.508490    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:03:16.508543 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.508499    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:03:16.508981 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.508963    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 17:03:16.508981 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.508976    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:03:16.509116 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.508979    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 17:03:16.509116 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.509040    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 17:03:16.509554 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.509541    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:03:16.509622 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.509567    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:03:16.509622 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.509580    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:03:16.510381 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.510321    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-80.ec2.internal.18c5488cb6020668  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-80.ec2.internal,UID:ip-10-0-140-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-80.ec2.internal,},FirstTimestamp:2026-07-24 17:03:16.508477032 +0000 UTC m=+0.339729712,LastTimestamp:2026-07-24 17:03:16.508477032 +0000 UTC m=+0.339729712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-80.ec2.internal,}"
Jul 24 17:03:16.511566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.511555    2563 policy_none.go:49] "None policy: Start"
Jul 24 17:03:16.511613 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.511570    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 17:03:16.511613 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.511582    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 17:03:16.513540 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.513530    2563 policy_none.go:47] "Start"
Jul 24 17:03:16.548235 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.548222    2563 manager.go:337] "Starting Device Plugin manager"
Jul 24 17:03:16.548298 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.548248    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 17:03:16.548298 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.548257    2563 server.go:87] "Starting device plugin registration server"
Jul 24 17:03:16.548493 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.548480    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 17:03:16.548542 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.548499    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 17:03:16.548580 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.548565    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 17:03:16.548758 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.548719    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 17:03:16.548758 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.548733    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 17:03:16.549131 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.549107    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 17:03:16.549209 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.549161    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:16.648851 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.648835    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:03:16.651019 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.649503    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:03:16.651019 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.649530    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:03:16.651019 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.649544    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:03:16.651019 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.649562    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.657607 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.657586    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.657607 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.657605    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-80.ec2.internal\": node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:16.674495 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.674449    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:16.680366 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.680348    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 17:03:16.681575 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.681553    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 17:03:16.681628 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.681594    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 17:03:16.681628 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.681612    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 17:03:16.681724 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.681641    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 17:03:16.683777 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.683762    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:03:16.775474 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.775446    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:16.782604 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.782586    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal"]
Jul 24 17:03:16.782674 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.782639    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:03:16.783415 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.783402    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:03:16.783462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.783427    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:03:16.783462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.783437    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:03:16.785839 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.785825    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:03:16.785955 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.785941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.785988 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.785981    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:03:16.786523 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.786508    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:03:16.786523 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.786517    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:03:16.786658 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.786533    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:03:16.786658 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.786538    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:03:16.786658 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.786547    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:03:16.786658 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.786554    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:03:16.788920 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.788906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.788973 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.788931    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:03:16.789536 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.789512    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:03:16.789625 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.789547    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:03:16.789625 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.789560    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:03:16.790888 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.790872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d562444ea96a0658e0ac8acab109779b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal\" (UID: \"d562444ea96a0658e0ac8acab109779b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.790949 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.790898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3526c73b5b056c8ae8b4457c3416ed52-config\") pod \"kube-apiserver-proxy-ip-10-0-140-80.ec2.internal\" (UID: \"3526c73b5b056c8ae8b4457c3416ed52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.790949 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.790915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d562444ea96a0658e0ac8acab109779b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal\" (UID: \"d562444ea96a0658e0ac8acab109779b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.816205 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.816186    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-80.ec2.internal\" not found" node="ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.820892 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.820876    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-80.ec2.internal\" not found" node="ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.876260 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.876233    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:16.891470 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.891444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d562444ea96a0658e0ac8acab109779b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal\" (UID: \"d562444ea96a0658e0ac8acab109779b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.891561 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.891476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d562444ea96a0658e0ac8acab109779b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal\" (UID: \"d562444ea96a0658e0ac8acab109779b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.891561 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.891501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3526c73b5b056c8ae8b4457c3416ed52-config\") pod \"kube-apiserver-proxy-ip-10-0-140-80.ec2.internal\" (UID: \"3526c73b5b056c8ae8b4457c3416ed52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.891561 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.891536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d562444ea96a0658e0ac8acab109779b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal\" (UID: \"d562444ea96a0658e0ac8acab109779b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.891561 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.891542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d562444ea96a0658e0ac8acab109779b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal\" (UID: \"d562444ea96a0658e0ac8acab109779b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.891561 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:16.891537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3526c73b5b056c8ae8b4457c3416ed52-config\") pod \"kube-apiserver-proxy-ip-10-0-140-80.ec2.internal\" (UID: \"3526c73b5b056c8ae8b4457c3416ed52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:16.977304 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:16.977246    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:17.077763 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:17.077734    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:17.119206 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.119182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:17.124442 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.124178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:17.178772 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:17.178740    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:17.279271 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:17.279218    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:17.379799 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:17.379780    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:17.432355 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.432338    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 17:03:17.432865 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.432472    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 17:03:17.480300 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:17.480271    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:17.505925 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.505890    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 16:58:16 +0000 UTC" deadline="2028-04-13 16:27:18.726893638 +0000 UTC"
Jul 24 17:03:17.505925 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.505920    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15095h24m1.22097681s"
Jul 24 17:03:17.580738 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:17.580677    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-80.ec2.internal\" not found"
Jul 24 17:03:17.590032 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.590014    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:03:17.689030 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.689005    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:17.701329 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.701309    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 17:03:17.703056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.703040    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal"
Jul 24 17:03:17.722320 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.722304    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 17:03:17.758088 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:17.758030    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3526c73b5b056c8ae8b4457c3416ed52.slice/crio-cc1b0f16b7d37844dc638d2651a1ec7a52de4f7dfe6db8c6faf093e25bfb86a6 WatchSource:0}: Error finding container cc1b0f16b7d37844dc638d2651a1ec7a52de4f7dfe6db8c6faf093e25bfb86a6: Status 404 returned error can't find the container with id cc1b0f16b7d37844dc638d2651a1ec7a52de4f7dfe6db8c6faf093e25bfb86a6
Jul 24 17:03:17.762620 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.762603    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:03:17.768824 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:17.768802    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd562444ea96a0658e0ac8acab109779b.slice/crio-2061b447094fcffec7c06dea90230328b0ccb2d75058812b02c3357d73a2c1d1 WatchSource:0}: Error finding container 2061b447094fcffec7c06dea90230328b0ccb2d75058812b02c3357d73a2c1d1: Status 404 returned error can't find the container with id 2061b447094fcffec7c06dea90230328b0ccb2d75058812b02c3357d73a2c1d1
Jul 24 17:03:17.786049 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.786033    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:03:17.869823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:17.869776    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:03:18.454995 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.454962    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:03:18.468564 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.468535    2563 apiserver.go:52] "Watching apiserver"
Jul 24 17:03:18.477541 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.477518    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 17:03:18.479511 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.479032    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-dpxpq","kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal","openshift-dns/node-resolver-mplnn","openshift-image-registry/node-ca-ztz7f","openshift-multus/multus-additional-cni-plugins-c7krh","openshift-multus/multus-swqzw","openshift-multus/network-metrics-daemon-w8hqg","openshift-network-diagnostics/network-check-target-h7zlm","kube-system/global-pull-secret-syncer-58fw9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx","openshift-cluster-node-tuning-operator/tuned-jfnxc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal","openshift-network-operator/iptables-alerter-7j94v","openshift-ovn-kubernetes/ovnkube-node-tjnf5"]
Jul 24 17:03:18.482222 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.481861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:18.482222 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.481972    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:18.484285 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.484260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.487808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.487161    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:03:18.487808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.487242    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 17:03:18.487808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.487461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.487808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.487473    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 17:03:18.488137 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.488120    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bsrkd\""
Jul 24 17:03:18.490326 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.489532    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 17:03:18.490326 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.489561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.490447 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.490360    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 17:03:18.491618 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.490654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 17:03:18.491618 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.490711    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xgzbb\""
Jul 24 17:03:18.491618 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.491376    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 17:03:18.491904 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.491627    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-99kcv\""
Jul 24 17:03:18.491904 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.491892    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 17:03:18.492166 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.492149    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 17:03:18.492575 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.492557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.494847 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.494821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.495045 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.495020    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 17:03:18.495142 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.495085    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 17:03:18.495230 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.495200    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dj8jg\""
Jul 24 17:03:18.495230 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.495218    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 17:03:18.495368 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.495309    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 17:03:18.495368 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.495330    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 17:03:18.496633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.496603    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:18.496747 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.496692    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:18.497832 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.497354    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 17:03:18.497832 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.497605    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5v2gv\""
Jul 24 17:03:18.499574 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.499553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:18.501146 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8353272a-9785-468a-896e-e7e6b13663ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.501247 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8353272a-9785-468a-896e-e7e6b13663ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.501247 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2cj\" (UniqueName: \"kubernetes.io/projected/852e6ad4-88e9-486e-962f-cfc44efa06ca-kube-api-access-fl2cj\") pod \"node-resolver-mplnn\" (UID: \"852e6ad4-88e9-486e-962f-cfc44efa06ca\") " pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.501247 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88c4f897-c3ac-4191-aee8-976fe6a42699-serviceca\") pod \"node-ca-ztz7f\" (UID: \"88c4f897-c3ac-4191-aee8-976fe6a42699\") " pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.501247 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b177c8c4-00ec-40b2-a253-ce061f049df6-host-slash\") pod \"iptables-alerter-7j94v\" (UID: \"b177c8c4-00ec-40b2-a253-ce061f049df6\") " pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/852e6ad4-88e9-486e-962f-cfc44efa06ca-hosts-file\") pod \"node-resolver-mplnn\" (UID: \"852e6ad4-88e9-486e-962f-cfc44efa06ca\") " pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/852e6ad4-88e9-486e-962f-cfc44efa06ca-tmp-dir\") pod \"node-resolver-mplnn\" (UID: \"852e6ad4-88e9-486e-962f-cfc44efa06ca\") " pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7fs\" (UniqueName: \"kubernetes.io/projected/8353272a-9785-468a-896e-e7e6b13663ec-kube-api-access-qr7fs\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpnjm\" (UniqueName: \"kubernetes.io/projected/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-kube-api-access-jpnjm\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b177c8c4-00ec-40b2-a253-ce061f049df6-iptables-alerter-script\") pod \"iptables-alerter-7j94v\" (UID: \"b177c8c4-00ec-40b2-a253-ce061f049df6\") " pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-system-cni-dir\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8353272a-9785-468a-896e-e7e6b13663ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88c4f897-c3ac-4191-aee8-976fe6a42699-host\") pod \"node-ca-ztz7f\" (UID: \"88c4f897-c3ac-4191-aee8-976fe6a42699\") " pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cm49\" (UniqueName: \"kubernetes.io/projected/b177c8c4-00ec-40b2-a253-ce061f049df6-kube-api-access-6cm49\") pod \"iptables-alerter-7j94v\" (UID: \"b177c8c4-00ec-40b2-a253-ce061f049df6\") " pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6cjd\" (UniqueName: \"kubernetes.io/projected/88c4f897-c3ac-4191-aee8-976fe6a42699-kube-api-access-t6cjd\") pod \"node-ca-ztz7f\" (UID: \"88c4f897-c3ac-4191-aee8-976fe6a42699\") " pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.501479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-cnibin\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.502137 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.501503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-os-release\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.502778 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.502756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p2jrb\""
Jul 24 17:03:18.502857 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.502796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.502857 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.502832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 17:03:18.503074 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.503060    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 17:03:18.505139 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.505120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.506159 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.506139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 17:03:18.506585 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.506181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 17:03:18.507661 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.507637    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.508739 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.508709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 17:03:18.508943 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.508926    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 17:03:18.509283 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.509262    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 17:03:18.509617 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.509597    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qxxlc\""
Jul 24 17:03:18.510043 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.510025    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 17:03:18.510114 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.510075    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 17:03:18.510450 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.510432    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 17:03:18.510512 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.510448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fwdmt\""
Jul 24 17:03:18.510701 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.510685    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 17:03:18.510835 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.510818    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 17:03:18.510895 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.510846    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 17:03:18.511303 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.511278    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5s8wd\""
Jul 24 17:03:18.513076 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.513046    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:03:18.514568 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.514533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:18.514706 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.514681    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:18.526722 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.526680    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xwdld"
Jul 24 17:03:18.536128 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.536106    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xwdld"
Jul 24 17:03:18.591722 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.591705    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-var-lib-cni-multus\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-etc-selinux\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-run-openvswitch\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-log-socket\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c-konnectivity-ca\") pod \"konnectivity-agent-dpxpq\" (UID: \"02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c\") " pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62539bf7-8276-46a7-a490-5ede01496cf4-env-overrides\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-sys-fs\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-tuned\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7fs\" (UniqueName: \"kubernetes.io/projected/8353272a-9785-468a-896e-e7e6b13663ec-kube-api-access-qr7fs\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpnjm\" (UniqueName: \"kubernetes.io/projected/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-kube-api-access-jpnjm\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88c4f897-c3ac-4191-aee8-976fe6a42699-host\") pod \"node-ca-ztz7f\" (UID: \"88c4f897-c3ac-4191-aee8-976fe6a42699\") " pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.603069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-dbus\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-kubelet\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-sysconfig\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cm49\" (UniqueName: \"kubernetes.io/projected/b177c8c4-00ec-40b2-a253-ce061f049df6-kube-api-access-6cm49\") pod \"iptables-alerter-7j94v\" (UID: \"b177c8c4-00ec-40b2-a253-ce061f049df6\") " pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-cni-dir\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-systemd\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-run\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-host\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fm8f\" (UniqueName: \"kubernetes.io/projected/b5de599c-f1dd-4fcb-949a-26cad80c1f34-kube-api-access-6fm8f\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-cnibin\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-os-release\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8353272a-9785-468a-896e-e7e6b13663ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88c4f897-c3ac-4191-aee8-976fe6a42699-serviceca\") pod \"node-ca-ztz7f\" (UID: \"88c4f897-c3ac-4191-aee8-976fe6a42699\") " pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.602976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02608831-5f3e-4349-b3bb-065f0cc65db5-cni-binary-copy\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-var-lib-cni-bin\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62539bf7-8276-46a7-a490-5ede01496cf4-ovnkube-config\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zck\" (UniqueName: \"kubernetes.io/projected/62539bf7-8276-46a7-a490-5ede01496cf4-kube-api-access-h8zck\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.604327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b177c8c4-00ec-40b2-a253-ce061f049df6-host-slash\") pod \"iptables-alerter-7j94v\" (UID: \"b177c8c4-00ec-40b2-a253-ce061f049df6\") " pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-etc-kubernetes\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ddb8\" (UniqueName: \"kubernetes.io/projected/02608831-5f3e-4349-b3bb-065f0cc65db5-kube-api-access-8ddb8\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-systemd-units\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-registration-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-device-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-sysctl-d\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8353272a-9785-468a-896e-e7e6b13663ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88c4f897-c3ac-4191-aee8-976fe6a42699-host\") pod \"node-ca-ztz7f\" (UID: \"88c4f897-c3ac-4191-aee8-976fe6a42699\") " pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b177c8c4-00ec-40b2-a253-ce061f049df6-host-slash\") pod \"iptables-alerter-7j94v\" (UID: \"b177c8c4-00ec-40b2-a253-ce061f049df6\") " pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-cnibin\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cjd\" (UniqueName: \"kubernetes.io/projected/88c4f897-c3ac-4191-aee8-976fe6a42699-kube-api-access-t6cjd\") pod \"node-ca-ztz7f\" (UID: \"88c4f897-c3ac-4191-aee8-976fe6a42699\") " pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-cnibin\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.603572    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.603669    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:19.103635941 +0000 UTC m=+2.934888632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:18.605385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.603574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-run-systemd\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8353272a-9785-468a-896e-e7e6b13663ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8353272a-9785-468a-896e-e7e6b13663ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-cni-bin\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-sys\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-var-lib-kubelet\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-conf-dir\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-daemon-config\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62539bf7-8276-46a7-a490-5ede01496cf4-ovnkube-script-lib\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88c4f897-c3ac-4191-aee8-976fe6a42699-serviceca\") pod \"node-ca-ztz7f\" (UID: \"88c4f897-c3ac-4191-aee8-976fe6a42699\") " pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-os-release\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8353272a-9785-468a-896e-e7e6b13663ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-run-k8s-cni-cncf-io\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.604963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-run-multus-certs\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c-agent-certs\") pod \"konnectivity-agent-dpxpq\" (UID: \"02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c\") " pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-slash\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.606231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-var-lib-openvswitch\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-node-log\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-var-lib-kubelet\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-socket-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5de599c-f1dd-4fcb-949a-26cad80c1f34-tmp\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-system-cni-dir\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-run-netns\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8353272a-9785-468a-896e-e7e6b13663ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62539bf7-8276-46a7-a490-5ede01496cf4-ovn-node-metrics-cert\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-sysctl-conf\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-os-release\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-hostroot\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-kubelet-config\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-etc-openvswitch\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2cj\" (UniqueName: \"kubernetes.io/projected/852e6ad4-88e9-486e-962f-cfc44efa06ca-kube-api-access-fl2cj\") pod \"node-resolver-mplnn\" (UID: \"852e6ad4-88e9-486e-962f-cfc44efa06ca\") " pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.607173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/852e6ad4-88e9-486e-962f-cfc44efa06ca-hosts-file\") pod \"node-resolver-mplnn\" (UID: \"852e6ad4-88e9-486e-962f-cfc44efa06ca\") " pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/852e6ad4-88e9-486e-962f-cfc44efa06ca-tmp-dir\") pod \"node-resolver-mplnn\" (UID: \"852e6ad4-88e9-486e-962f-cfc44efa06ca\") " pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-run-ovn\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-cni-netd\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8vb\" (UniqueName: \"kubernetes.io/projected/decd8615-1d72-47e7-b3d7-6c32f88fafb8-kube-api-access-6c8vb\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-modprobe-d\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-kubernetes\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-lib-modules\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b177c8c4-00ec-40b2-a253-ce061f049df6-iptables-alerter-script\") pod \"iptables-alerter-7j94v\" (UID: \"b177c8c4-00ec-40b2-a253-ce061f049df6\") " pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-system-cni-dir\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-socket-dir-parent\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-run-netns\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/852e6ad4-88e9-486e-962f-cfc44efa06ca-hosts-file\") pod \"node-resolver-mplnn\" (UID: \"852e6ad4-88e9-486e-962f-cfc44efa06ca\") " pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.605943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8353272a-9785-468a-896e-e7e6b13663ec-system-cni-dir\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.606185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/852e6ad4-88e9-486e-962f-cfc44efa06ca-tmp-dir\") pod \"node-resolver-mplnn\" (UID: \"852e6ad4-88e9-486e-962f-cfc44efa06ca\") " pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.608165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.606195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b177c8c4-00ec-40b2-a253-ce061f049df6-iptables-alerter-script\") pod \"iptables-alerter-7j94v\" (UID: \"b177c8c4-00ec-40b2-a253-ce061f049df6\") " pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.611770 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.611744    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 17:03:18.624743 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.616655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7fs\" (UniqueName: \"kubernetes.io/projected/8353272a-9785-468a-896e-e7e6b13663ec-kube-api-access-qr7fs\") pod \"multus-additional-cni-plugins-c7krh\" (UID: \"8353272a-9785-468a-896e-e7e6b13663ec\") " pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.624743 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.616783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpnjm\" (UniqueName: \"kubernetes.io/projected/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-kube-api-access-jpnjm\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:18.624743 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.617975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cm49\" (UniqueName: \"kubernetes.io/projected/b177c8c4-00ec-40b2-a253-ce061f049df6-kube-api-access-6cm49\") pod \"iptables-alerter-7j94v\" (UID: \"b177c8c4-00ec-40b2-a253-ce061f049df6\") " pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.624743 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.618381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cjd\" (UniqueName: \"kubernetes.io/projected/88c4f897-c3ac-4191-aee8-976fe6a42699-kube-api-access-t6cjd\") pod \"node-ca-ztz7f\" (UID: \"88c4f897-c3ac-4191-aee8-976fe6a42699\") " pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.624743 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.618922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2cj\" (UniqueName: \"kubernetes.io/projected/852e6ad4-88e9-486e-962f-cfc44efa06ca-kube-api-access-fl2cj\") pod \"node-resolver-mplnn\" (UID: \"852e6ad4-88e9-486e-962f-cfc44efa06ca\") " pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.692088 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.692021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal" event={"ID":"d562444ea96a0658e0ac8acab109779b","Type":"ContainerStarted","Data":"2061b447094fcffec7c06dea90230328b0ccb2d75058812b02c3357d73a2c1d1"}
Jul 24 17:03:18.695144 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.695114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal" event={"ID":"3526c73b5b056c8ae8b4457c3416ed52","Type":"ContainerStarted","Data":"cc1b0f16b7d37844dc638d2651a1ec7a52de4f7dfe6db8c6faf093e25bfb86a6"}
Jul 24 17:03:18.706837 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-run-netns\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.706837 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62539bf7-8276-46a7-a490-5ede01496cf4-ovn-node-metrics-cert\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.706837 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-sysctl-conf\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.706837 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-os-release\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-hostroot\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-kubelet-config\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-etc-openvswitch\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-run-ovn\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-cni-netd\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8vb\" (UniqueName: \"kubernetes.io/projected/decd8615-1d72-47e7-b3d7-6c32f88fafb8-kube-api-access-6c8vb\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-modprobe-d\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.706999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-kubernetes\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-lib-modules\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-socket-dir-parent\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.707064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-run-netns\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-var-lib-cni-multus\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-etc-selinux\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-run-openvswitch\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-log-socket\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c-konnectivity-ca\") pod \"konnectivity-agent-dpxpq\" (UID: \"02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c\") " pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62539bf7-8276-46a7-a490-5ede01496cf4-env-overrides\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-sys-fs\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-tuned\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-dbus\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-kubelet\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-sysconfig\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-cni-dir\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-systemd\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-run\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-host\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.707633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fm8f\" (UniqueName: \"kubernetes.io/projected/b5de599c-f1dd-4fcb-949a-26cad80c1f34-kube-api-access-6fm8f\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02608831-5f3e-4349-b3bb-065f0cc65db5-cni-binary-copy\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-var-lib-cni-bin\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62539bf7-8276-46a7-a490-5ede01496cf4-ovnkube-config\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zck\" (UniqueName: \"kubernetes.io/projected/62539bf7-8276-46a7-a490-5ede01496cf4-kube-api-access-h8zck\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-etc-kubernetes\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ddb8\" (UniqueName: \"kubernetes.io/projected/02608831-5f3e-4349-b3bb-065f0cc65db5-kube-api-access-8ddb8\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-systemd-units\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-registration-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-device-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-sysctl-d\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-cnibin\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-run-systemd\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-cni-bin\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-sys\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-var-lib-kubelet\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-conf-dir\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.708443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-daemon-config\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62539bf7-8276-46a7-a490-5ede01496cf4-ovnkube-script-lib\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-run-k8s-cni-cncf-io\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-run-multus-certs\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.707985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c-agent-certs\") pod \"konnectivity-agent-dpxpq\" (UID: \"02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c\") " pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-slash\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-var-lib-openvswitch\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-node-log\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-var-lib-kubelet\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-socket-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5de599c-f1dd-4fcb-949a-26cad80c1f34-tmp\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-system-cni-dir\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62539bf7-8276-46a7-a490-5ede01496cf4-env-overrides\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-run-netns\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-system-cni-dir\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.708358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-sys-fs\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.709358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.709322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-dbus\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:18.710346 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.709383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-kubelet\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.710346 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.709436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-sysconfig\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.710346 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.709499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-cni-dir\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.710346 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.709549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-systemd\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.710346 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.709599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-run\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.710346 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.709648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-host\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.710558 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.710389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02608831-5f3e-4349-b3bb-065f0cc65db5-cni-binary-copy\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.710558 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.710463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-var-lib-cni-bin\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.710985 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.710960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62539bf7-8276-46a7-a490-5ede01496cf4-ovnkube-config\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62539bf7-8276-46a7-a490-5ede01496cf4-ovn-node-metrics-cert\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-kubernetes\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-modprobe-d\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-lib-modules\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-sysctl-conf\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-var-lib-cni-multus\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-run-netns\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-tuned\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-run-openvswitch\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-log-socket\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-run-ovn\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-os-release\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-hostroot\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-etc-openvswitch\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-cni-netd\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-conf-dir\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.719069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-kubelet-config\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.711853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-etc-kubernetes\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-socket-dir-parent\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c-konnectivity-ca\") pod \"konnectivity-agent-dpxpq\" (UID: \"02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c\") " pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-systemd-units\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-etc-selinux\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-slash\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-run-systemd\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-sys\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-var-lib-kubelet\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.712303    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-registration-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.712389    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret podName:da9a090f-6bf3-4c56-87c9-c48b9fa615d4 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:19.212369535 +0000 UTC m=+3.043622211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret") pod "global-pull-secret-syncer-58fw9" (UID: "da9a090f-6bf3-4c56-87c9-c48b9fa615d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-cnibin\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5de599c-f1dd-4fcb-949a-26cad80c1f34-etc-sysctl-d\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-cni-bin\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.719954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-device-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-var-lib-kubelet\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-node-log\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-run-multus-certs\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/decd8615-1d72-47e7-b3d7-6c32f88fafb8-socket-dir\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02608831-5f3e-4349-b3bb-065f0cc65db5-host-run-k8s-cni-cncf-io\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62539bf7-8276-46a7-a490-5ede01496cf4-ovnkube-script-lib\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.712975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62539bf7-8276-46a7-a490-5ede01496cf4-var-lib-openvswitch\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.713349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02608831-5f3e-4349-b3bb-065f0cc65db5-multus-daemon-config\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.718466    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.718494    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.718511    2563 projected.go:196] Error preparing data for projected volume kube-api-access-6grv6 for pod openshift-network-diagnostics/network-check-target-h7zlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:18.720578 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:18.718585    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6 podName:2a545805-cc97-489f-af65-4244ac30f3b6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:19.218564968 +0000 UTC m=+3.049817664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6grv6" (UniqueName: "kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6") pod "network-check-target-h7zlm" (UID: "2a545805-cc97-489f-af65-4244ac30f3b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:18.730887 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.730821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zck\" (UniqueName: \"kubernetes.io/projected/62539bf7-8276-46a7-a490-5ede01496cf4-kube-api-access-h8zck\") pod \"ovnkube-node-tjnf5\" (UID: \"62539bf7-8276-46a7-a490-5ede01496cf4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.731322 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.731279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c-agent-certs\") pod \"konnectivity-agent-dpxpq\" (UID: \"02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c\") " pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:18.731411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.731338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fm8f\" (UniqueName: \"kubernetes.io/projected/b5de599c-f1dd-4fcb-949a-26cad80c1f34-kube-api-access-6fm8f\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.731523 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.731501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5de599c-f1dd-4fcb-949a-26cad80c1f34-tmp\") pod \"tuned-jfnxc\" (UID: \"b5de599c-f1dd-4fcb-949a-26cad80c1f34\") " pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:18.732697 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.732669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ddb8\" (UniqueName: \"kubernetes.io/projected/02608831-5f3e-4349-b3bb-065f0cc65db5-kube-api-access-8ddb8\") pod \"multus-swqzw\" (UID: \"02608831-5f3e-4349-b3bb-065f0cc65db5\") " pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.733584 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.733566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8vb\" (UniqueName: \"kubernetes.io/projected/decd8615-1d72-47e7-b3d7-6c32f88fafb8-kube-api-access-6c8vb\") pod \"aws-ebs-csi-driver-node-p7hqx\" (UID: \"decd8615-1d72-47e7-b3d7-6c32f88fafb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.802847 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.802814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7j94v"
Jul 24 17:03:18.819070 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.819038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mplnn"
Jul 24 17:03:18.830100 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.830076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztz7f"
Jul 24 17:03:18.839271 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.839247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c7krh"
Jul 24 17:03:18.847329 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.847311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swqzw"
Jul 24 17:03:18.854625 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.854604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:18.863874 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.863814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:18.872588 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.872571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx"
Jul 24 17:03:18.877409 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:18.877386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jfnxc"
Jul 24 17:03:19.111285 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:19.111202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:19.111460 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:19.111385    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:19.111460 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:19.111445    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:20.111425376 +0000 UTC m=+3.942678046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:19.312491 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:19.312459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:19.312670 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:19.312500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:19.312670 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:19.312626    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:03:19.312670 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:19.312643    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:03:19.312670 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:19.312652    2563 projected.go:196] Error preparing data for projected volume kube-api-access-6grv6 for pod openshift-network-diagnostics/network-check-target-h7zlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:19.312859 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:19.312724    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6 podName:2a545805-cc97-489f-af65-4244ac30f3b6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:20.312704113 +0000 UTC m=+4.143956782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6grv6" (UniqueName: "kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6") pod "network-check-target-h7zlm" (UID: "2a545805-cc97-489f-af65-4244ac30f3b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:19.313041 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:19.313016    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:19.313166 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:19.313080    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret podName:da9a090f-6bf3-4c56-87c9-c48b9fa615d4 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:20.313062461 +0000 UTC m=+4.144315127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret") pod "global-pull-secret-syncer-58fw9" (UID: "da9a090f-6bf3-4c56-87c9-c48b9fa615d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:19.539893 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:19.539853    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 16:58:18 +0000 UTC" deadline="2028-05-06 01:10:19.649614918 +0000 UTC"
Jul 24 17:03:19.539893 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:19.539890    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15632h7m0.109729792s"
Jul 24 17:03:19.681970 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:19.681944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:19.682114 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:19.682064    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:19.881417 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:19.881385    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5de599c_f1dd_4fcb_949a_26cad80c1f34.slice/crio-54724f385e0e6011a6ae2360a57e4b538e114118564ef1a7b8458fcd3b579c10 WatchSource:0}: Error finding container 54724f385e0e6011a6ae2360a57e4b538e114118564ef1a7b8458fcd3b579c10: Status 404 returned error can't find the container with id 54724f385e0e6011a6ae2360a57e4b538e114118564ef1a7b8458fcd3b579c10
Jul 24 17:03:19.917471 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:19.917442    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecd8615_1d72_47e7_b3d7_6c32f88fafb8.slice/crio-9abdae1ce85c6f9752b4854a6a3ccdba3c71ca51475333b6fc435844989252f0 WatchSource:0}: Error finding container 9abdae1ce85c6f9752b4854a6a3ccdba3c71ca51475333b6fc435844989252f0: Status 404 returned error can't find the container with id 9abdae1ce85c6f9752b4854a6a3ccdba3c71ca51475333b6fc435844989252f0
Jul 24 17:03:19.918259 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:19.918204    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62539bf7_8276_46a7_a490_5ede01496cf4.slice/crio-7a01335febddb86889e39c865628ec592c9cb0054026bd1813daf29bb5c40328 WatchSource:0}: Error finding container 7a01335febddb86889e39c865628ec592c9cb0054026bd1813daf29bb5c40328: Status 404 returned error can't find the container with id 7a01335febddb86889e39c865628ec592c9cb0054026bd1813daf29bb5c40328
Jul 24 17:03:19.919002 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:19.918981    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ca41c2_e80b_4342_bc4b_93b2ccf4ed0c.slice/crio-e77332b6c06f5b9625a6a1c7341f155cb690625bb341f449820f4b0d67822cc5 WatchSource:0}: Error finding container e77332b6c06f5b9625a6a1c7341f155cb690625bb341f449820f4b0d67822cc5: Status 404 returned error can't find the container with id e77332b6c06f5b9625a6a1c7341f155cb690625bb341f449820f4b0d67822cc5
Jul 24 17:03:19.920492 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:19.920449    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02608831_5f3e_4349_b3bb_065f0cc65db5.slice/crio-d980aa3d64bf331c8611711aca629e84bf1fe9d91c22bae3c33abb147de315ce WatchSource:0}: Error finding container d980aa3d64bf331c8611711aca629e84bf1fe9d91c22bae3c33abb147de315ce: Status 404 returned error can't find the container with id d980aa3d64bf331c8611711aca629e84bf1fe9d91c22bae3c33abb147de315ce
Jul 24 17:03:20.126623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.126477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:20.126720 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.126630    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:20.126720 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.126698    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:22.126683155 +0000 UTC m=+5.957935823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:20.328027 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.327997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:20.328152 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.328061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:20.328204 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.328178    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:20.328204 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.328186    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:03:20.328310 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.328208    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:03:20.328310 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.328220    2563 projected.go:196] Error preparing data for projected volume kube-api-access-6grv6 for pod openshift-network-diagnostics/network-check-target-h7zlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:20.328310 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.328245    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret podName:da9a090f-6bf3-4c56-87c9-c48b9fa615d4 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:22.328224054 +0000 UTC m=+6.159476724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret") pod "global-pull-secret-syncer-58fw9" (UID: "da9a090f-6bf3-4c56-87c9-c48b9fa615d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:20.328455 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.328332    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6 podName:2a545805-cc97-489f-af65-4244ac30f3b6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:22.328309321 +0000 UTC m=+6.159561987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6grv6" (UniqueName: "kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6") pod "network-check-target-h7zlm" (UID: "2a545805-cc97-489f-af65-4244ac30f3b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:20.540970 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.540819    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 16:58:18 +0000 UTC" deadline="2028-01-13 05:42:10.490679222 +0000 UTC"
Jul 24 17:03:20.540970 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.540884    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12900h38m49.94979983s"
Jul 24 17:03:20.685290 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.684545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:20.685290 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.684678    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:20.685290 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.685116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:20.685290 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:20.685215    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:20.711226 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.711192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swqzw" event={"ID":"02608831-5f3e-4349-b3bb-065f0cc65db5","Type":"ContainerStarted","Data":"d980aa3d64bf331c8611711aca629e84bf1fe9d91c22bae3c33abb147de315ce"}
Jul 24 17:03:20.713920 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.713888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dpxpq" event={"ID":"02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c","Type":"ContainerStarted","Data":"e77332b6c06f5b9625a6a1c7341f155cb690625bb341f449820f4b0d67822cc5"}
Jul 24 17:03:20.719552 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.719522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerStarted","Data":"7a01335febddb86889e39c865628ec592c9cb0054026bd1813daf29bb5c40328"}
Jul 24 17:03:20.723037 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.723007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx" event={"ID":"decd8615-1d72-47e7-b3d7-6c32f88fafb8","Type":"ContainerStarted","Data":"9abdae1ce85c6f9752b4854a6a3ccdba3c71ca51475333b6fc435844989252f0"}
Jul 24 17:03:20.725379 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.725352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jfnxc" event={"ID":"b5de599c-f1dd-4fcb-949a-26cad80c1f34","Type":"ContainerStarted","Data":"54724f385e0e6011a6ae2360a57e4b538e114118564ef1a7b8458fcd3b579c10"}
Jul 24 17:03:20.733626 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.733596    2563 generic.go:356] "Generic (PLEG): container finished" podID="d562444ea96a0658e0ac8acab109779b" containerID="e9327f2a14318c267ec5ac98507aef889ad860d5671cf98216cd0e41d0a83908" exitCode=0
Jul 24 17:03:20.733755 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.733680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal" event={"ID":"d562444ea96a0658e0ac8acab109779b","Type":"ContainerDied","Data":"e9327f2a14318c267ec5ac98507aef889ad860d5671cf98216cd0e41d0a83908"}
Jul 24 17:03:20.764345 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.764314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7j94v" event={"ID":"b177c8c4-00ec-40b2-a253-ce061f049df6","Type":"ContainerStarted","Data":"1598805d7a914c64b7e4c066661a98d185086225bf906f91b2e5bfd24a362bce"}
Jul 24 17:03:20.785495 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.785465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c7krh" event={"ID":"8353272a-9785-468a-896e-e7e6b13663ec","Type":"ContainerStarted","Data":"6abe8a4385cbb81816e3c785c65df96f393de85216ad16140461c8eb29b3bc46"}
Jul 24 17:03:20.789303 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.789278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mplnn" event={"ID":"852e6ad4-88e9-486e-962f-cfc44efa06ca","Type":"ContainerStarted","Data":"c8b82856d58a8597df6ea1a1d15d322d880da719a9fe80bdc98e08bb839959af"}
Jul 24 17:03:20.804892 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.804813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztz7f" event={"ID":"88c4f897-c3ac-4191-aee8-976fe6a42699","Type":"ContainerStarted","Data":"f78338ac44e3892b53ff2d867d79236424a96f728c8ee235db81bff9e296f0fa"}
Jul 24 17:03:20.825504 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:20.825472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal" event={"ID":"3526c73b5b056c8ae8b4457c3416ed52","Type":"ContainerStarted","Data":"91854e09388202534cef2ada6525f2d6a49e98ead09c79e384fd5381d94cd187"}
Jul 24 17:03:21.682120 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:21.682005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:21.682683 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:21.682158    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:21.844872 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:21.844787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal" event={"ID":"d562444ea96a0658e0ac8acab109779b","Type":"ContainerStarted","Data":"ac0a62f655a21e5f2e9eca8f6f7c988b2b7d0de946a6df542ccd58867ccf9e68"}
Jul 24 17:03:21.861249 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:21.861181    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-80.ec2.internal" podStartSLOduration=3.8611605730000003 podStartE2EDuration="3.861160573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:20.840355463 +0000 UTC m=+4.671608155" watchObservedRunningTime="2026-07-24 17:03:21.861160573 +0000 UTC m=+5.692413265"
Jul 24 17:03:22.143424 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:22.143340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:22.143597 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.143516    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:22.143597 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.143577    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:26.143559303 +0000 UTC m=+9.974811976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:22.345358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:22.344354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:22.345358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:22.344412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:22.345358 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.344561    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:03:22.345358 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.344579    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:03:22.345358 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.344593    2563 projected.go:196] Error preparing data for projected volume kube-api-access-6grv6 for pod openshift-network-diagnostics/network-check-target-h7zlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:22.345358 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.344677    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6 podName:2a545805-cc97-489f-af65-4244ac30f3b6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:26.34465696 +0000 UTC m=+10.175909633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6grv6" (UniqueName: "kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6") pod "network-check-target-h7zlm" (UID: "2a545805-cc97-489f-af65-4244ac30f3b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:22.345358 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.345210    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:22.345358 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.345293    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret podName:da9a090f-6bf3-4c56-87c9-c48b9fa615d4 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:26.345271455 +0000 UTC m=+10.176524122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret") pod "global-pull-secret-syncer-58fw9" (UID: "da9a090f-6bf3-4c56-87c9-c48b9fa615d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:22.683325 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:22.682743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:22.683325 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.682863    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:22.683325 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:22.683157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:22.683325 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:22.683252    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:23.682408 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:23.681905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:23.682408 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:23.682048    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:24.684632 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:24.684596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:24.685071 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:24.684728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:24.685071 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:24.684806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:24.685071 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:24.684871    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:25.682428 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:25.682396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:25.682631 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:25.682602    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:26.179706 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:26.179608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:26.180073 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.179823    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:26.180073 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.179885    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:34.179867023 +0000 UTC m=+18.011119697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:26.381717 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:26.381682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:26.381873 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:26.381732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:26.381873 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.381866    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:03:26.381971 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.381886    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:03:26.381971 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.381901    2563 projected.go:196] Error preparing data for projected volume kube-api-access-6grv6 for pod openshift-network-diagnostics/network-check-target-h7zlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:26.381971 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.381955    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6 podName:2a545805-cc97-489f-af65-4244ac30f3b6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:34.381937247 +0000 UTC m=+18.213189914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6grv6" (UniqueName: "kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6") pod "network-check-target-h7zlm" (UID: "2a545805-cc97-489f-af65-4244ac30f3b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:26.382150 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.382037    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:26.382150 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.382073    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret podName:da9a090f-6bf3-4c56-87c9-c48b9fa615d4 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:34.38206112 +0000 UTC m=+18.213313790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret") pod "global-pull-secret-syncer-58fw9" (UID: "da9a090f-6bf3-4c56-87c9-c48b9fa615d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:26.683356 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:26.683325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:26.683509 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.683424    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:26.683589 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:26.683520    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:26.683665 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:26.683623    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:27.682378 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:27.682342    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:27.682837 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:27.682474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:28.682832 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:28.682756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:28.683248 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:28.682875    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:28.683248 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:28.682763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:28.683248 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:28.683026    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:29.682378 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:29.682347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:29.682541 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:29.682507    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:30.682442 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:30.682410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:30.682904 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:30.682502    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:30.682904 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:30.682582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:30.682904 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:30.682692    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:31.682226 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:31.682193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:31.682405 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:31.682318    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:32.682004 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:32.681970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:32.682004 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:32.681985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:32.682439 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:32.682074    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:32.682439 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:32.682221    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:33.681836 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:33.681798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:33.682067 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:33.681931    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:34.244601 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:34.244567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:34.244894 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.244706    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:34.244894 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.244774    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:50.244754386 +0000 UTC m=+34.076007056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:34.445531 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:34.445496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:34.445715 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:34.445589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:34.445715 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.445686    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:34.445801 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.445727    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:03:34.445801 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.445749    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:03:34.445801 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.445758    2563 projected.go:196] Error preparing data for projected volume kube-api-access-6grv6 for pod openshift-network-diagnostics/network-check-target-h7zlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:34.445801 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.445732    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret podName:da9a090f-6bf3-4c56-87c9-c48b9fa615d4 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:50.445719877 +0000 UTC m=+34.276972541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret") pod "global-pull-secret-syncer-58fw9" (UID: "da9a090f-6bf3-4c56-87c9-c48b9fa615d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:34.445925 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.445805    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6 podName:2a545805-cc97-489f-af65-4244ac30f3b6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:50.445793761 +0000 UTC m=+34.277046428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6grv6" (UniqueName: "kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6") pod "network-check-target-h7zlm" (UID: "2a545805-cc97-489f-af65-4244ac30f3b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:34.682270 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:34.682203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:34.682270 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:34.682221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:34.682729 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.682327    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:34.682729 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:34.682420    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:35.682484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:35.682450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:35.682962 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:35.682583    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:36.683310 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:36.683277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:36.683798 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:36.683352    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:36.683798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:36.683722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:36.683911 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:36.683803    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:37.681820 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:37.681785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:37.682010 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:37.681917    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:38.681995 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:38.681956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:38.681995 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:38.681990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:38.682505 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:38.682058    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:38.682505 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:38.682197    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:39.682330 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.682070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:39.683067 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:39.682405    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:39.881565 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.881521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx" event={"ID":"decd8615-1d72-47e7-b3d7-6c32f88fafb8","Type":"ContainerStarted","Data":"3b1317e77ebab2648737bcd41eee13977f317b98a70b69ea2b442c3a93d6e034"}
Jul 24 17:03:39.883705 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.883272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jfnxc" event={"ID":"b5de599c-f1dd-4fcb-949a-26cad80c1f34","Type":"ContainerStarted","Data":"4d4fb0af471aca179f271ea7c6db7e4134bf650fc8837a7b253c7d2f21675d02"}
Jul 24 17:03:39.884876 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.884845    2563 generic.go:356] "Generic (PLEG): container finished" podID="8353272a-9785-468a-896e-e7e6b13663ec" containerID="bbc6ccf11d4efa0f893c641b3759abcc617cf4bcdf21c9bfc73e276003eb75e7" exitCode=0
Jul 24 17:03:39.885008 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.884934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c7krh" event={"ID":"8353272a-9785-468a-896e-e7e6b13663ec","Type":"ContainerDied","Data":"bbc6ccf11d4efa0f893c641b3759abcc617cf4bcdf21c9bfc73e276003eb75e7"}
Jul 24 17:03:39.889538 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.889511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mplnn" event={"ID":"852e6ad4-88e9-486e-962f-cfc44efa06ca","Type":"ContainerStarted","Data":"8d0059bd46e2a06679b2b051a7f0181c41a7199c2a9c814b6143e07040032d3c"}
Jul 24 17:03:39.891183 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.891162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztz7f" event={"ID":"88c4f897-c3ac-4191-aee8-976fe6a42699","Type":"ContainerStarted","Data":"a487a16d0106942f092a13133b35e877c43db98efb56b9ac502594b0fa0303fe"}
Jul 24 17:03:39.892927 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.892903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swqzw" event={"ID":"02608831-5f3e-4349-b3bb-065f0cc65db5","Type":"ContainerStarted","Data":"5dfcc7cd8974665e3fe0d8bd5ea927b298a52571e7b76c6093fc95a64bc14d5e"}
Jul 24 17:03:39.894797 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.894737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dpxpq" event={"ID":"02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c","Type":"ContainerStarted","Data":"8464ed43a791b07dce0f96bb97edd4fa56a09d1406433beff1b95ae4f881be62"}
Jul 24 17:03:39.898819 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.898801    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:03:39.899143 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.899126    2563 generic.go:356] "Generic (PLEG): container finished" podID="62539bf7-8276-46a7-a490-5ede01496cf4" containerID="c43a110d3ba9e40665c7cff820fddd1b168db339cd61dac765fe66c131fb8591" exitCode=1
Jul 24 17:03:39.899218 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.899151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerStarted","Data":"a019b3e5951c68d4f42aa847644db8abd26b4657aec009feae253486e1674ab8"}
Jul 24 17:03:39.899218 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.899204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerStarted","Data":"3839c3c7f0cf273cc4ee6adc7c133f66ea00c9a5d8ef8914d92d1d5dfaf9a156"}
Jul 24 17:03:39.899286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.899236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerStarted","Data":"34b27f1d70b4fcb124eac731931f26672495f3d16045fd86f81547dafd76ce9b"}
Jul 24 17:03:39.899286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.899263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerStarted","Data":"961370afca448bc7bdcea20fb824fa80307e9f4f536fd422d4b10a4d01d17398"}
Jul 24 17:03:39.899384 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.899309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerDied","Data":"c43a110d3ba9e40665c7cff820fddd1b168db339cd61dac765fe66c131fb8591"}
Jul 24 17:03:39.899384 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.899341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerStarted","Data":"1c2e8464b2a5920d4c58bcdb0f2cb3aa918918500001439743639da356fc148b"}
Jul 24 17:03:39.900275 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.900243    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jfnxc" podStartSLOduration=3.978236055 podStartE2EDuration="22.900232274s" totalImagesPullingTime="18.921996219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:39.89997347 +0000 UTC m=+23.731226150" watchObservedRunningTime="2026-07-24 17:03:39.900232274 +0000 UTC m=+23.731484961"
Jul 24 17:03:39.900356 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.900327    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-80.ec2.internal" podStartSLOduration=21.900321198 podStartE2EDuration="21.900321198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:21.861771931 +0000 UTC m=+5.693024622" watchObservedRunningTime="2026-07-24 17:03:39.900321198 +0000 UTC m=+23.731573884"
Jul 24 17:03:39.918595 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.918552    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dpxpq" podStartSLOduration=4.097878291 podStartE2EDuration="22.918539716s" totalImagesPullingTime="18.820661425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:39.918142233 +0000 UTC m=+23.749394914" watchObservedRunningTime="2026-07-24 17:03:39.918539716 +0000 UTC m=+23.749792403"
Jul 24 17:03:39.932819 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.932778    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ztz7f" podStartSLOduration=4.01636513 podStartE2EDuration="22.932767164s" totalImagesPullingTime="18.916402034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:39.932505192 +0000 UTC m=+23.763757896" watchObservedRunningTime="2026-07-24 17:03:39.932767164 +0000 UTC m=+23.764019850"
Jul 24 17:03:39.948873 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:39.948838    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mplnn" podStartSLOduration=4.032809877 podStartE2EDuration="22.94881263s" totalImagesPullingTime="18.916002753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:39.948613879 +0000 UTC m=+23.779866567" watchObservedRunningTime="2026-07-24 17:03:39.94881263 +0000 UTC m=+23.780065318"
Jul 24 17:03:40.000301 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.000265    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-swqzw" podStartSLOduration=4.087794907 podStartE2EDuration="23.000254184s" totalImagesPullingTime="18.912459277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:39.99950359 +0000 UTC m=+23.830756276" watchObservedRunningTime="2026-07-24 17:03:40.000254184 +0000 UTC m=+23.831506910"
Jul 24 17:03:40.112283 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.112256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:40.182431 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.182405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:40.270186 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.270113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:40.332534 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.332505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:40.493919 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.493898    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 17:03:40.570021 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.569839    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T17:03:40.49391437Z","UUID":"1dcaef0a-ece6-4070-b32a-c581b460dfb9","Handler":null,"Name":"","Endpoint":""}
Jul 24 17:03:40.571341 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.571319    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 17:03:40.571447 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.571353    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 17:03:40.681856 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.681828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:40.681997 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:40.681945    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:40.682069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.682006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:40.682120 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:40.682096    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:40.903692 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.903600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx" event={"ID":"decd8615-1d72-47e7-b3d7-6c32f88fafb8","Type":"ContainerStarted","Data":"1571add65d02c286f66a73ec114011e5b0293495922d7bcf9753f2d8fd6ac792"}
Jul 24 17:03:40.905273 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.905245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7j94v" event={"ID":"b177c8c4-00ec-40b2-a253-ce061f049df6","Type":"ContainerStarted","Data":"2a39ce3220d82a0ccafcb3b3018f6231acafe613e05955ea740b431856ee6a6c"}
Jul 24 17:03:40.919603 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:40.919559    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7j94v" podStartSLOduration=5.090607192 podStartE2EDuration="23.919546404s" totalImagesPullingTime="18.828939212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:40.91914004 +0000 UTC m=+24.750392728" watchObservedRunningTime="2026-07-24 17:03:40.919546404 +0000 UTC m=+24.750799090"
Jul 24 17:03:41.061437 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:41.061403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:41.115168 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:41.115137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:41.477507 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:41.477475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:41.529415 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:41.529382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:41.682267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:41.682239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:41.682424 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:41.682341    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:42.268197 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:42.268006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:42.665368 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:42.665296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:42.682267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:42.682241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:42.682420 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:42.682252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:42.682420 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:42.682361    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:42.682529 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:42.682423    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:42.765623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:42.765589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:42.913087 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:42.913056    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:03:42.913468 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:42.913435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerStarted","Data":"f84ee7bf745067de6e6d0f1d71e6da924eba272f4e5e0517d73f812d67b103c5"}
Jul 24 17:03:42.916032 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:42.915974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx" event={"ID":"decd8615-1d72-47e7-b3d7-6c32f88fafb8","Type":"ContainerStarted","Data":"1177dc399ba0fca807c84080f3d161ff64d4544c7a687b9ab66a66945f8e2c0b"}
Jul 24 17:03:42.933862 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:42.933808    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7hqx" podStartSLOduration=4.593563741 podStartE2EDuration="25.933791524s" totalImagesPullingTime="21.340227783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:42.933286747 +0000 UTC m=+26.764539435" watchObservedRunningTime="2026-07-24 17:03:42.933791524 +0000 UTC m=+26.765044213"
Jul 24 17:03:43.123803 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:43.123772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:43.316143 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:43.316118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:43.316852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:43.316828    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:43.337798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:43.337773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:43.396093 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:43.396069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:43.553378 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:43.553348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:43.633809 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:43.633730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:43.682521 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:43.682497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:43.682685 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:43.682611    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:44.078265 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.078235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:44.180418 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.180389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:44.682739 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.682565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:44.683200 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.682626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:44.683200 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:44.682801    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:44.683200 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:44.682872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:44.924080 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.924029    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:03:44.924416 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.924394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerStarted","Data":"33ae0952f99ccc02f6059bc0b602a2dd8658ed4c864ae484494c8f820162b0ca"}
Jul 24 17:03:44.924641 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.924625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:44.924882 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.924864    2563 scope.go:117] "RemoveContainer" containerID="c43a110d3ba9e40665c7cff820fddd1b168db339cd61dac765fe66c131fb8591"
Jul 24 17:03:44.926412 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.926392    2563 generic.go:356] "Generic (PLEG): container finished" podID="8353272a-9785-468a-896e-e7e6b13663ec" containerID="8dc312611d06dac02608e88f55b9c24fabca1e173b5f0cc1408f70b7d5df39c8" exitCode=0
Jul 24 17:03:44.926516 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.926421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c7krh" event={"ID":"8353272a-9785-468a-896e-e7e6b13663ec","Type":"ContainerDied","Data":"8dc312611d06dac02608e88f55b9c24fabca1e173b5f0cc1408f70b7d5df39c8"}
Jul 24 17:03:44.940007 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:44.939990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:45.682111 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:45.682081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:45.682302 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:45.682207    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:45.933588 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:45.933535    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:03:45.933966 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:45.933886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" event={"ID":"62539bf7-8276-46a7-a490-5ede01496cf4","Type":"ContainerStarted","Data":"1695f3f32ae8bb56f8672b12429076c550269d30cf0c1a990fbf182d0cfdc078"}
Jul 24 17:03:45.934090 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:45.934072    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 17:03:45.934346 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:45.934329    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:45.951385 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:45.951360    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:45.967524 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:45.967470    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5" podStartSLOduration=9.776059269 podStartE2EDuration="28.967450448s" totalImagesPullingTime="19.191391179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:45.965554154 +0000 UTC m=+29.796806843" watchObservedRunningTime="2026-07-24 17:03:45.967450448 +0000 UTC m=+29.798703135"
Jul 24 17:03:46.290341 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.290172    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:46.290478 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.290454    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 17:03:46.290805 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.290785    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dpxpq"
Jul 24 17:03:46.299986 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.299958    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-58fw9"]
Jul 24 17:03:46.300089 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.300066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:46.300189 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:46.300168    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:46.303092 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.303069    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8hqg"]
Jul 24 17:03:46.303190 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.303175    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:46.303301 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:46.303280    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:46.303695 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.303674    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h7zlm"]
Jul 24 17:03:46.303793 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.303756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:46.303862 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:46.303844    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:46.356736 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.356711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:46.937699 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.937665    2563 generic.go:356] "Generic (PLEG): container finished" podID="8353272a-9785-468a-896e-e7e6b13663ec" containerID="ad9cb5f21c997056c8d5f0a0b24ef4ff06aff19b7a4a2614d69905a95ea95a38" exitCode=0
Jul 24 17:03:46.938142 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.937746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c7krh" event={"ID":"8353272a-9785-468a-896e-e7e6b13663ec","Type":"ContainerDied","Data":"ad9cb5f21c997056c8d5f0a0b24ef4ff06aff19b7a4a2614d69905a95ea95a38"}
Jul 24 17:03:46.938142 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:46.937847    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 17:03:47.480406 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:47.480370    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:03:47.682085 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:47.682057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:47.682258 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:47.682057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:47.682258 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:47.682186    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:47.682258 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:47.682219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:48.682250 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:48.682174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:48.682848 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:48.682294    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:48.944229 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:48.944165    2563 generic.go:356] "Generic (PLEG): container finished" podID="8353272a-9785-468a-896e-e7e6b13663ec" containerID="07cb3d01e55c2d3c53cd6dde6bdbae8959cc9e24ca9a51da6f94b5d19dd0f2b1" exitCode=0
Jul 24 17:03:48.944350 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:48.944254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c7krh" event={"ID":"8353272a-9785-468a-896e-e7e6b13663ec","Type":"ContainerDied","Data":"07cb3d01e55c2d3c53cd6dde6bdbae8959cc9e24ca9a51da6f94b5d19dd0f2b1"}
Jul 24 17:03:49.681813 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:49.681780    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:49.681813 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:49.681795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:49.682062 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:49.681898    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:49.682062 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:49.682035    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:50.267978 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:50.267941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:50.268494 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:50.268120    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:50.268494 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:50.268196    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:22.268173784 +0000 UTC m=+66.099426452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:03:50.469584 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:50.469556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:50.469787 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:50.469616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:50.469787 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:50.469726    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:50.469787 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:50.469741    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:03:50.469787 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:50.469761    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:03:50.469787 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:50.469776    2563 projected.go:196] Error preparing data for projected volume kube-api-access-6grv6 for pod openshift-network-diagnostics/network-check-target-h7zlm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:50.469787 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:50.469779    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret podName:da9a090f-6bf3-4c56-87c9-c48b9fa615d4 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:22.469763366 +0000 UTC m=+66.301016034 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret") pod "global-pull-secret-syncer-58fw9" (UID: "da9a090f-6bf3-4c56-87c9-c48b9fa615d4") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:03:50.470141 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:50.469823    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6 podName:2a545805-cc97-489f-af65-4244ac30f3b6 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:22.469808018 +0000 UTC m=+66.301060707 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6grv6" (UniqueName: "kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6") pod "network-check-target-h7zlm" (UID: "2a545805-cc97-489f-af65-4244ac30f3b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:03:50.682959 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:50.682535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:50.682959 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:50.682630    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7zlm" podUID="2a545805-cc97-489f-af65-4244ac30f3b6"
Jul 24 17:03:51.682693 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:51.682641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:51.683153 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:51.682642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:51.683153 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:51.682814    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:03:51.683153 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:51.682860    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58fw9" podUID="da9a090f-6bf3-4c56-87c9-c48b9fa615d4"
Jul 24 17:03:52.056897 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.056869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:52.129928 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.129893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:52.348189 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.348116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:52.453441 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.453409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:52.564849 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.564819    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-80.ec2.internal" event="NodeReady"
Jul 24 17:03:52.565000 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.564976    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 17:03:52.609895 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.609805    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zk69n"]
Jul 24 17:03:52.615819 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.615152    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w25vs"]
Jul 24 17:03:52.615819 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.615245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.618214 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.618189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-57f2l\""
Jul 24 17:03:52.618613 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.618597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w25vs"
Jul 24 17:03:52.619317 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.619299    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 17:03:52.619522 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.619497    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 17:03:52.620866 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.620849    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 17:03:52.621042 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.621022    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 17:03:52.621343 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.621323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c58hb\""
Jul 24 17:03:52.621572 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.621551    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 17:03:52.624063 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.624039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w25vs"]
Jul 24 17:03:52.627544 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.627521    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zk69n"]
Jul 24 17:03:52.681977 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.681948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:03:52.684714 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.684683    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2xn4k\""
Jul 24 17:03:52.685067 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.684719    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 17:03:52.685067 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.684721    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 17:03:52.732327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.732270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:52.785399 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.785369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.785399 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.785410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwc6d\" (UniqueName: \"kubernetes.io/projected/d8201c68-e268-411e-886e-87a4c1429052-kube-api-access-dwc6d\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.785577 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.785457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cec7ee9b-390e-407d-a9a6-7ff6d615867a-cert\") pod \"ingress-canary-w25vs\" (UID: \"cec7ee9b-390e-407d-a9a6-7ff6d615867a\") " pod="openshift-ingress-canary/ingress-canary-w25vs"
Jul 24 17:03:52.785577 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.785481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8201c68-e268-411e-886e-87a4c1429052-tmp-dir\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.785577 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.785536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d8201c68-e268-411e-886e-87a4c1429052-config-volume\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.785577 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.785571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtd9l\" (UniqueName: \"kubernetes.io/projected/cec7ee9b-390e-407d-a9a6-7ff6d615867a-kube-api-access-rtd9l\") pod \"ingress-canary-w25vs\" (UID: \"cec7ee9b-390e-407d-a9a6-7ff6d615867a\") " pod="openshift-ingress-canary/ingress-canary-w25vs"
Jul 24 17:03:52.791905 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.791862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:52.886139 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.886060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cec7ee9b-390e-407d-a9a6-7ff6d615867a-cert\") pod \"ingress-canary-w25vs\" (UID: \"cec7ee9b-390e-407d-a9a6-7ff6d615867a\") " pod="openshift-ingress-canary/ingress-canary-w25vs"
Jul 24 17:03:52.886139 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.886105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8201c68-e268-411e-886e-87a4c1429052-tmp-dir\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.886337 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.886308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d8201c68-e268-411e-886e-87a4c1429052-config-volume\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.886469 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.886371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtd9l\" (UniqueName: \"kubernetes.io/projected/cec7ee9b-390e-407d-a9a6-7ff6d615867a-kube-api-access-rtd9l\") pod \"ingress-canary-w25vs\" (UID: \"cec7ee9b-390e-407d-a9a6-7ff6d615867a\") " pod="openshift-ingress-canary/ingress-canary-w25vs"
Jul 24 17:03:52.886469 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.886418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.886469 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.886442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8201c68-e268-411e-886e-87a4c1429052-tmp-dir\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.886627 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.886443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwc6d\" (UniqueName: \"kubernetes.io/projected/d8201c68-e268-411e-886e-87a4c1429052-kube-api-access-dwc6d\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.886707 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:52.886635    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:03:52.886773 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:52.886712    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls podName:d8201c68-e268-411e-886e-87a4c1429052 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:53.386693495 +0000 UTC m=+37.217946165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls") pod "dns-default-zk69n" (UID: "d8201c68-e268-411e-886e-87a4c1429052") : secret "dns-default-metrics-tls" not found
Jul 24 17:03:52.887232 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.887207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d8201c68-e268-411e-886e-87a4c1429052-config-volume\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.891039 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.891017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cec7ee9b-390e-407d-a9a6-7ff6d615867a-cert\") pod \"ingress-canary-w25vs\" (UID: \"cec7ee9b-390e-407d-a9a6-7ff6d615867a\") " pod="openshift-ingress-canary/ingress-canary-w25vs"
Jul 24 17:03:52.895828 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.895805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwc6d\" (UniqueName: \"kubernetes.io/projected/d8201c68-e268-411e-886e-87a4c1429052-kube-api-access-dwc6d\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:52.896559 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.896537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtd9l\" (UniqueName: \"kubernetes.io/projected/cec7ee9b-390e-407d-a9a6-7ff6d615867a-kube-api-access-rtd9l\") pod \"ingress-canary-w25vs\" (UID: \"cec7ee9b-390e-407d-a9a6-7ff6d615867a\") " pod="openshift-ingress-canary/ingress-canary-w25vs"
Jul 24 17:03:52.936603 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:52.936577    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w25vs"
Jul 24 17:03:53.272383 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.272230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w25vs"]
Jul 24 17:03:53.276008 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:03:53.275984    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec7ee9b_390e_407d_a9a6_7ff6d615867a.slice/crio-8a2403399e50b9ecb0258348c0de65380e7da34725cf3c5549493a7ff7856db8 WatchSource:0}: Error finding container 8a2403399e50b9ecb0258348c0de65380e7da34725cf3c5549493a7ff7856db8: Status 404 returned error can't find the container with id 8a2403399e50b9ecb0258348c0de65380e7da34725cf3c5549493a7ff7856db8
Jul 24 17:03:53.390113 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.390049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:53.390211 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:53.390193    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:03:53.390269 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:53.390260    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls podName:d8201c68-e268-411e-886e-87a4c1429052 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:54.390243468 +0000 UTC m=+38.221496133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls") pod "dns-default-zk69n" (UID: "d8201c68-e268-411e-886e-87a4c1429052") : secret "dns-default-metrics-tls" not found
Jul 24 17:03:53.682145 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.682075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:03:53.682281 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.682235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:03:53.684740 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.684708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 17:03:53.684740 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.684739    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 17:03:53.685154 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.684708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pgbn9\""
Jul 24 17:03:53.959429 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.959347    2563 generic.go:356] "Generic (PLEG): container finished" podID="8353272a-9785-468a-896e-e7e6b13663ec" containerID="274b05ee24d241a767ebc627aa03c00f841e629291e2c53c1576ca302d954885" exitCode=0
Jul 24 17:03:53.959586 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.959441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c7krh" event={"ID":"8353272a-9785-468a-896e-e7e6b13663ec","Type":"ContainerDied","Data":"274b05ee24d241a767ebc627aa03c00f841e629291e2c53c1576ca302d954885"}
Jul 24 17:03:53.960719 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:53.960570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w25vs" event={"ID":"cec7ee9b-390e-407d-a9a6-7ff6d615867a","Type":"ContainerStarted","Data":"8a2403399e50b9ecb0258348c0de65380e7da34725cf3c5549493a7ff7856db8"}
Jul 24 17:03:54.397805 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:54.397773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:54.397973 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:54.397933    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:03:54.398051 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:54.398026    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls podName:d8201c68-e268-411e-886e-87a4c1429052 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:56.39800103 +0000 UTC m=+40.229253711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls") pod "dns-default-zk69n" (UID: "d8201c68-e268-411e-886e-87a4c1429052") : secret "dns-default-metrics-tls" not found
Jul 24 17:03:54.967200 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:54.967167    2563 generic.go:356] "Generic (PLEG): container finished" podID="8353272a-9785-468a-896e-e7e6b13663ec" containerID="2bdc855b687807dcf7e22b7e33d7f840aaa05159a5ff6592129556e376b08a64" exitCode=0
Jul 24 17:03:54.967595 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:54.967212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c7krh" event={"ID":"8353272a-9785-468a-896e-e7e6b13663ec","Type":"ContainerDied","Data":"2bdc855b687807dcf7e22b7e33d7f840aaa05159a5ff6592129556e376b08a64"}
Jul 24 17:03:55.971572 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:55.971386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w25vs" event={"ID":"cec7ee9b-390e-407d-a9a6-7ff6d615867a","Type":"ContainerStarted","Data":"c0f9c489c9da52e1cf1b7e6e09f4eb9ecccedc3345f4e5ef3ede78d5452dbe9d"}
Jul 24 17:03:55.975566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:55.975536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c7krh" event={"ID":"8353272a-9785-468a-896e-e7e6b13663ec","Type":"ContainerStarted","Data":"0498e9386e705562828e2329c4fd9f558ea0f10cb6817b56a8faa73b94dc9728"}
Jul 24 17:03:55.988884 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:55.988842    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w25vs" podStartSLOduration=1.4445028180000001 podStartE2EDuration="3.988829467s" totalImagesPullingTime="2.544326649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:55.988529774 +0000 UTC m=+39.819782466" watchObservedRunningTime="2026-07-24 17:03:55.988829467 +0000 UTC m=+39.820082154"
Jul 24 17:03:56.011206 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:56.011144    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c7krh" podStartSLOduration=8.978623233 podStartE2EDuration="39.011105428s" totalImagesPullingTime="30.032482195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:56.010590093 +0000 UTC m=+39.841842774" watchObservedRunningTime="2026-07-24 17:03:56.011105428 +0000 UTC m=+39.842358116"
Jul 24 17:03:56.097768 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:56.097687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:56.157576 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:56.157547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:56.310391 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:56.310367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:56.378595 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:56.378539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:56.412924 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:56.412898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:03:56.413045 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:56.413028    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:03:56.413108 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:03:56.413098    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls podName:d8201c68-e268-411e-886e-87a4c1429052 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:00.413081184 +0000 UTC m=+44.244333851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls") pod "dns-default-zk69n" (UID: "d8201c68-e268-411e-886e-87a4c1429052") : secret "dns-default-metrics-tls" not found
Jul 24 17:03:56.632881 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:56.632807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:03:56.686740 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:03:56.686712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:04:00.437131 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:00.437091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:04:00.437622 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:00.437281    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:04:00.437622 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:00.437367    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls podName:d8201c68-e268-411e-886e-87a4c1429052 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:08.43734556 +0000 UTC m=+52.268598228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls") pod "dns-default-zk69n" (UID: "d8201c68-e268-411e-886e-87a4c1429052") : secret "dns-default-metrics-tls" not found
Jul 24 17:04:04.423927 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:04.423887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:04:04.481293 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:04.481254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:04:08.489192 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:08.489159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:04:08.489584 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:08.489287    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:04:08.489584 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:08.489342    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls podName:d8201c68-e268-411e-886e-87a4c1429052 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:24.489327075 +0000 UTC m=+68.320579741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls") pod "dns-default-zk69n" (UID: "d8201c68-e268-411e-886e-87a4c1429052") : secret "dns-default-metrics-tls" not found
Jul 24 17:04:18.202582 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.202547    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"]
Jul 24 17:04:18.204383 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.204345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:04:18.237630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.237603    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.237630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.237615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"]
Jul 24 17:04:18.239963 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.239938    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 24 17:04:18.239963 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.239936    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 17:04:18.240223 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.240198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 17:04:18.240223 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.240194    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 24 17:04:18.240317 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.240259    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 17:04:18.240742 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.240725    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 24 17:04:18.240840 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.240794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 24 17:04:18.254829 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.254809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.254923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.254844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-hub\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.254923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.254863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5e99fb-8044-4e81-a95e-295982f1fa88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.254923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.254878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.254923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.254894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-ca\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.255072 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.254949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq8mk\" (UniqueName: \"kubernetes.io/projected/0b5e99fb-8044-4e81-a95e-295982f1fa88-kube-api-access-cq8mk\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.268824 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.268805    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:04:18.355414 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.355388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.355501 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.355427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-hub\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.355501 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.355445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5e99fb-8044-4e81-a95e-295982f1fa88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.355501 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.355468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.355501 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.355493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-ca\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.355683 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.355605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8mk\" (UniqueName: \"kubernetes.io/projected/0b5e99fb-8044-4e81-a95e-295982f1fa88-kube-api-access-cq8mk\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.356130 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.356111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5e99fb-8044-4e81-a95e-295982f1fa88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.359412 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.359393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-ca\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.359530 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.359508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.359581 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.359567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.359628 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.359611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b5e99fb-8044-4e81-a95e-295982f1fa88-hub\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.363119 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.363092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq8mk\" (UniqueName: \"kubernetes.io/projected/0b5e99fb-8044-4e81-a95e-295982f1fa88-kube-api-access-cq8mk\") pod \"cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8\" (UID: \"0b5e99fb-8044-4e81-a95e-295982f1fa88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.563312 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.563285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:04:18.686147 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.686121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"]
Jul 24 17:04:18.689211 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:04:18.689187    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b5e99fb_8044_4e81_a95e_295982f1fa88.slice/crio-9aa3e272156ea93a1d09d37b382d6ae9d96843f3271a3eabfdaa357c0fe55e7f WatchSource:0}: Error finding container 9aa3e272156ea93a1d09d37b382d6ae9d96843f3271a3eabfdaa357c0fe55e7f: Status 404 returned error can't find the container with id 9aa3e272156ea93a1d09d37b382d6ae9d96843f3271a3eabfdaa357c0fe55e7f
Jul 24 17:04:18.955760 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:18.955701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjnf5"
Jul 24 17:04:19.034861 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:19.034831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" event={"ID":"0b5e99fb-8044-4e81-a95e-295982f1fa88","Type":"ContainerStarted","Data":"9aa3e272156ea93a1d09d37b382d6ae9d96843f3271a3eabfdaa357c0fe55e7f"}
Jul 24 17:04:19.078512 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:19.078485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:04:19.183127 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:19.183103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:04:22.283520 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.283496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:04:22.285486 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.285470    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 17:04:22.294316 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:22.294296    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 17:04:22.294404 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:22.294353    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:05:26.294335429 +0000 UTC m=+130.125588098 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : secret "metrics-daemon-secret" not found
Jul 24 17:04:22.484849 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.484825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:04:22.484954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.484919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:04:22.488056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.488039    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 17:04:22.488105 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.488082    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 17:04:22.497959 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.497945    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 17:04:22.498210 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.498192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da9a090f-6bf3-4c56-87c9-c48b9fa615d4-original-pull-secret\") pod \"global-pull-secret-syncer-58fw9\" (UID: \"da9a090f-6bf3-4c56-87c9-c48b9fa615d4\") " pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:04:22.509005 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.508984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grv6\" (UniqueName: \"kubernetes.io/projected/2a545805-cc97-489f-af65-4244ac30f3b6-kube-api-access-6grv6\") pod \"network-check-target-h7zlm\" (UID: \"2a545805-cc97-489f-af65-4244ac30f3b6\") " pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:04:22.697365 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.697309    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2xn4k\""
Jul 24 17:04:22.706048 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.706020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:04:22.798206 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.798181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58fw9"
Jul 24 17:04:22.820209 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.820174    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h7zlm"]
Jul 24 17:04:22.823591 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:04:22.823568    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a545805_cc97_489f_af65_4244ac30f3b6.slice/crio-4e114749165c64ffc118f1af8652ad96bb5b7102ffb63c7be888d8e5ba6ff01f WatchSource:0}: Error finding container 4e114749165c64ffc118f1af8652ad96bb5b7102ffb63c7be888d8e5ba6ff01f: Status 404 returned error can't find the container with id 4e114749165c64ffc118f1af8652ad96bb5b7102ffb63c7be888d8e5ba6ff01f
Jul 24 17:04:22.929200 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:22.929170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-58fw9"]
Jul 24 17:04:22.932114 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:04:22.932086    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9a090f_6bf3_4c56_87c9_c48b9fa615d4.slice/crio-cd01af0ef8b300f0e2439ba2c218398ab05fea2a26c8fe45ffa94d9a39eff3ce WatchSource:0}: Error finding container cd01af0ef8b300f0e2439ba2c218398ab05fea2a26c8fe45ffa94d9a39eff3ce: Status 404 returned error can't find the container with id cd01af0ef8b300f0e2439ba2c218398ab05fea2a26c8fe45ffa94d9a39eff3ce
Jul 24 17:04:23.045745 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:23.045709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h7zlm" event={"ID":"2a545805-cc97-489f-af65-4244ac30f3b6","Type":"ContainerStarted","Data":"4e114749165c64ffc118f1af8652ad96bb5b7102ffb63c7be888d8e5ba6ff01f"}
Jul 24 17:04:23.047769 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:23.047745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" event={"ID":"0b5e99fb-8044-4e81-a95e-295982f1fa88","Type":"ContainerStarted","Data":"c133acf0ca3cbdc50dfd12f2ac5a7087a68c0d24b4ee805f89a989acf6461d71"}
Jul 24 17:04:23.047878 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:23.047782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" event={"ID":"0b5e99fb-8044-4e81-a95e-295982f1fa88","Type":"ContainerStarted","Data":"a691b4fa78a026425325c33b9cde9a2b348bb890db0ed3ab0ed3954624153442"}
Jul 24 17:04:23.047878 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:23.047793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" event={"ID":"0b5e99fb-8044-4e81-a95e-295982f1fa88","Type":"ContainerStarted","Data":"dfdf7813a2edbc1c33b3203213abc34d31ca4e230789001295ad31f904cf79e1"}
Jul 24 17:04:23.048830 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:23.048813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-58fw9" event={"ID":"da9a090f-6bf3-4c56-87c9-c48b9fa615d4","Type":"ContainerStarted","Data":"cd01af0ef8b300f0e2439ba2c218398ab05fea2a26c8fe45ffa94d9a39eff3ce"}
Jul 24 17:04:23.064766 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:23.064728    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" podStartSLOduration=1.478182673 podStartE2EDuration="5.064713983s" totalImagesPullingTime="3.58653131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:23.063539607 +0000 UTC m=+66.894792293" watchObservedRunningTime="2026-07-24 17:04:23.064713983 +0000 UTC m=+66.895966672"
Jul 24 17:04:24.498734 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:24.498694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:04:24.499238 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:24.498828    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:04:24.499238 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:24.498888    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls podName:d8201c68-e268-411e-886e-87a4c1429052 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:56.498874066 +0000 UTC m=+100.330126731 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls") pod "dns-default-zk69n" (UID: "d8201c68-e268-411e-886e-87a4c1429052") : secret "dns-default-metrics-tls" not found
Jul 24 17:04:28.065316 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:28.065273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h7zlm" event={"ID":"2a545805-cc97-489f-af65-4244ac30f3b6","Type":"ContainerStarted","Data":"f05fe03d62d8f2d0fc1fe5af59482aa19d2ec5ef1d67c0d62377cb6f50c1e0b2"}
Jul 24 17:04:28.065937 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:28.065388    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:04:28.067145 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:28.067121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-58fw9" event={"ID":"da9a090f-6bf3-4c56-87c9-c48b9fa615d4","Type":"ContainerStarted","Data":"c9821ef5640cc67434702d54171d5b87b1dbbc376efd297a1e362567d6e58d5f"}
Jul 24 17:04:28.081205 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:28.081144    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h7zlm" podStartSLOduration=66.456167378 podStartE2EDuration="1m11.081132597s" totalImagesPullingTime="4.624965219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:28.080072409 +0000 UTC m=+71.911325097" watchObservedRunningTime="2026-07-24 17:04:28.081132597 +0000 UTC m=+71.912385275"
Jul 24 17:04:28.093415 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:28.093367    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-58fw9" podStartSLOduration=66.572252665 podStartE2EDuration="1m11.093353333s" totalImagesPullingTime="4.521100668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:28.093081772 +0000 UTC m=+71.924334458" watchObservedRunningTime="2026-07-24 17:04:28.093353333 +0000 UTC m=+71.924606019"
Jul 24 17:04:56.513765 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:56.513720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:04:56.514192 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:56.513877    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:04:56.514192 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:04:56.513946    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls podName:d8201c68-e268-411e-886e-87a4c1429052 nodeName:}" failed. No retries permitted until 2026-07-24 17:06:00.513931824 +0000 UTC m=+164.345184489 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls") pod "dns-default-zk69n" (UID: "d8201c68-e268-411e-886e-87a4c1429052") : secret "dns-default-metrics-tls" not found
Jul 24 17:04:59.072750 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:04:59.072716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h7zlm"
Jul 24 17:05:26.316894 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:05:26.316860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:05:26.317376 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:05:26.316977    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 17:05:26.317376 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:05:26.317029    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:28.317014531 +0000 UTC m=+252.148267196 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : secret "metrics-daemon-secret" not found
Jul 24 17:05:55.629331 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:05:55.629284    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zk69n" podUID="d8201c68-e268-411e-886e-87a4c1429052"
Jul 24 17:05:56.307620 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:05:56.307592    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zk69n"
Jul 24 17:05:56.692738 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:05:56.692659    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:06:00.548708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:06:00.548637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:06:00.549095 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:06:00.548808    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 17:06:00.549095 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:06:00.548871    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls podName:d8201c68-e268-411e-886e-87a4c1429052 nodeName:}" failed. No retries permitted until 2026-07-24 17:08:02.548855447 +0000 UTC m=+286.380108113 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls") pod "dns-default-zk69n" (UID: "d8201c68-e268-411e-886e-87a4c1429052") : secret "dns-default-metrics-tls" not found
Jul 24 17:06:10.682507 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:06:10.682427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:06:18.493865 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:06:18.493835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:06:41.632054 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:06:41.632015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:15.573449 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:15.573412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:15.670927 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:15.670897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:17.715092 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:17.715061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:17.774006 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:17.773984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:19.374462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.374428    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pb8t2"]
Jul 24 17:07:19.377451 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.377432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.379604 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.379563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 17:07:19.379881 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.379851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 17:07:19.379979 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.379883    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 17:07:19.379979 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.379880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 17:07:19.380381 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.380367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2qh5k\""
Jul 24 17:07:19.395715 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.395697    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pb8t2"]
Jul 24 17:07:19.424800 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.424774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:19.459941 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.459917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.460030 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.459960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jf9\" (UniqueName: \"kubernetes.io/projected/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-api-access-h6jf9\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.460030 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.460011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.460133 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.460037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/726e53f0-e3d8-4414-929b-683f5f4eb5c2-crio-socket\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.460133 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.460060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/726e53f0-e3d8-4414-929b-683f5f4eb5c2-data-volume\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.486402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.486378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:19.560499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.560482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.560597 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.560514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jf9\" (UniqueName: \"kubernetes.io/projected/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-api-access-h6jf9\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.560597 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.560548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.560597 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.560574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/726e53f0-e3d8-4414-929b-683f5f4eb5c2-crio-socket\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.560737 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.560634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/726e53f0-e3d8-4414-929b-683f5f4eb5c2-crio-socket\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.560737 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:19.560636    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:19.560737 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.560675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/726e53f0-e3d8-4414-929b-683f5f4eb5c2-data-volume\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.560737 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:19.560735    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:20.060718537 +0000 UTC m=+243.891971205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:19.560888 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:19.560662    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:07:19.560888 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:19.560799    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:20.060782451 +0000 UTC m=+243.892035116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : configmap "kube-rbac-proxy" not found
Jul 24 17:07:19.560993 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.560977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/726e53f0-e3d8-4414-929b-683f5f4eb5c2-data-volume\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.570743 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.570721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jf9\" (UniqueName: \"kubernetes.io/projected/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-api-access-h6jf9\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:19.639502 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.639449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:19.739128 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:19.739097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:20.064507 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:20.064484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:20.064616 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:20.064541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:20.064616 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:20.064595    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:07:20.064722 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:20.064640    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:20.064722 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:20.064713    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:21.064696883 +0000 UTC m=+244.895949548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:20.064831 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:20.064730    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:21.064721101 +0000 UTC m=+244.895973767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : configmap "kube-rbac-proxy" not found
Jul 24 17:07:20.418917 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:20.418843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:20.512091 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:20.512067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:20.614608 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:20.614580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:20.710865 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:20.710812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:20.778615 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:20.778595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:21.071363 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:21.071345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:21.071476 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:21.071388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:21.071476 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:21.071457    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:07:21.071476 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:21.071475    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:21.071569 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:21.071518    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:23.071506755 +0000 UTC m=+246.902759420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:21.071569 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:21.071532    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:23.071524593 +0000 UTC m=+246.902777258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : configmap "kube-rbac-proxy" not found
Jul 24 17:07:21.361133 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:21.361071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:21.476311 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:21.476285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:21.537207 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:21.537189    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mplnn_852e6ad4-88e9-486e-962f-cfc44efa06ca/dns-node-resolver/0.log"
Jul 24 17:07:22.533712 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:22.533689    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztz7f_88c4f897-c3ac-4191-aee8-976fe6a42699/node-ca/0.log"
Jul 24 17:07:22.969334 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:22.969306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:23.071352 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:23.071327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:23.083462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:23.083441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:23.083519 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:23.083486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:23.083564 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:23.083548    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:07:23.083601 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:23.083572    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:23.083640 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:23.083599    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:27.083584473 +0000 UTC m=+250.914837138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : configmap "kube-rbac-proxy" not found
Jul 24 17:07:23.083640 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:23.083614    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:27.08360736 +0000 UTC m=+250.914860024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:23.133087 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:23.133064    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w25vs_cec7ee9b-390e-407d-a9a6-7ff6d615867a/serve-healthcheck-canary/0.log"
Jul 24 17:07:24.039169 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:24.039138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:24.128256 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:24.128226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:24.198458 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:24.198403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:24.410391 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:24.410322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:24.508712 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:24.508689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:24.609363 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:24.609336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:24.709235 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:24.709181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:24.810659 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:24.810621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:25.516328 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:25.516296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:26.092331 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:26.092299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:26.305940 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:26.305913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:26.495641 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:26.495613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:26.691410 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:26.691385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:27.112677 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:27.112616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:27.112830 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:27.112701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:27.112830 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:27.112733    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 17:07:27.112830 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:27.112781    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:27.112830 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:27.112806    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:35.112788768 +0000 UTC m=+258.944041436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : configmap "kube-rbac-proxy" not found
Jul 24 17:07:27.112830 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:27.112825    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls podName:726e53f0-e3d8-4414-929b-683f5f4eb5c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:35.112813128 +0000 UTC m=+258.944065793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb8t2" (UID: "726e53f0-e3d8-4414-929b-683f5f4eb5c2") : secret "insights-runtime-extractor-tls" not found
Jul 24 17:07:27.494109 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:27.494077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:27.651239 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:27.651218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:27.951403 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:27.951319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:28.100689 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:28.100662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:28.304630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:28.304586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:28.320787 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:28.320764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:07:28.320903 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:28.320887    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 17:07:28.320972 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:28.320964    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs podName:07768a9d-923d-4e34-9c7d-4db82a6cf6c2 nodeName:}" failed. No retries permitted until 2026-07-24 17:09:30.320945753 +0000 UTC m=+374.152198422 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs") pod "network-metrics-daemon-w8hqg" (UID: "07768a9d-923d-4e34-9c7d-4db82a6cf6c2") : secret "metrics-daemon-secret" not found
Jul 24 17:07:28.373398 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:28.373376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:28.691278 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:28.691256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:28.893891 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:28.893829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:29.100553 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:29.100527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:30.095365 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:30.095336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:30.493933 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:30.493903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:30.688169 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:30.688139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:30.894075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:30.893999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:30.947718 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:30.947688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:31.094096 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:31.094072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:32.098856 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:32.098820    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:32.291923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:32.291900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:32.493618 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:32.493591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:32.658074 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:32.658044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:32.897686 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:32.897588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:33.579111 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:33.579081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:34.011568 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:34.011537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:34.305545 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:34.305468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:34.694480 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:34.694414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:34.891776 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:34.891748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:35.065923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:35.065895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:35.130765 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:35.130737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:35.170817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:35.170796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:35.170874 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:35.170845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:35.171318 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:35.171293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/726e53f0-e3d8-4414-929b-683f5f4eb5c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:35.173170 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:35.173143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/726e53f0-e3d8-4414-929b-683f5f4eb5c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb8t2\" (UID: \"726e53f0-e3d8-4414-929b-683f5f4eb5c2\") " pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:35.286438 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:35.286418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pb8t2"
Jul 24 17:07:35.408298 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:35.408265    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pb8t2"]
Jul 24 17:07:35.412242 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:07:35.412210    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726e53f0_e3d8_4414_929b_683f5f4eb5c2.slice/crio-eaf96bbfd87556d0ac5d58a04c1135bd8d7adc09922ad7561ce53fba757f888f WatchSource:0}: Error finding container eaf96bbfd87556d0ac5d58a04c1135bd8d7adc09922ad7561ce53fba757f888f: Status 404 returned error can't find the container with id eaf96bbfd87556d0ac5d58a04c1135bd8d7adc09922ad7561ce53fba757f888f
Jul 24 17:07:35.574345 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:35.574314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pb8t2" event={"ID":"726e53f0-e3d8-4414-929b-683f5f4eb5c2","Type":"ContainerStarted","Data":"eaf96bbfd87556d0ac5d58a04c1135bd8d7adc09922ad7561ce53fba757f888f"}
Jul 24 17:07:37.038787 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:37.038708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:37.101075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:37.101036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:39.868778 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:39.868748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:39.942040 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:39.942011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:40.639835 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:40.639813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:41.075824 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:41.075772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:41.175888 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:41.175854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:41.513477 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:41.513450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:41.594151 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:41.594118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pb8t2" event={"ID":"726e53f0-e3d8-4414-929b-683f5f4eb5c2","Type":"ContainerStarted","Data":"99edc9946d04ba2e1d8b18d28487177074ccf1e9f43139e6fc74c8f6e9a926fc"}
Jul 24 17:07:41.613891 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:41.613861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:41.722377 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:41.722339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:42.139316 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:42.139280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:42.190261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:42.190214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:42.322398 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:42.322358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:42.523347 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:42.523315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:42.624021 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:42.623989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:44.606109 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:44.606076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pb8t2" event={"ID":"726e53f0-e3d8-4414-929b-683f5f4eb5c2","Type":"ContainerStarted","Data":"9527aa9e8d59d5b6a4cf1cb159015583dfa5f6b0eaafc975a284d1908bac1397"}
Jul 24 17:07:47.132237 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.132201    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-r2v9x"]
Jul 24 17:07:47.156100 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.156071    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-r2v9x"]
Jul 24 17:07:47.156251 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.156189    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-r2v9x"
Jul 24 17:07:47.158497 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.158473    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9mrjp\""
Jul 24 17:07:47.158497 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.158486    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 17:07:47.158712 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.158553    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 17:07:47.160160 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.160136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468pm\" (UniqueName: \"kubernetes.io/projected/99ba6f44-50af-47c9-9fec-370390ebfd14-kube-api-access-468pm\") pod \"downloads-7c97dc4555-r2v9x\" (UID: \"99ba6f44-50af-47c9-9fec-370390ebfd14\") " pod="openshift-console/downloads-7c97dc4555-r2v9x"
Jul 24 17:07:47.260969 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.260937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468pm\" (UniqueName: \"kubernetes.io/projected/99ba6f44-50af-47c9-9fec-370390ebfd14-kube-api-access-468pm\") pod \"downloads-7c97dc4555-r2v9x\" (UID: \"99ba6f44-50af-47c9-9fec-370390ebfd14\") " pod="openshift-console/downloads-7c97dc4555-r2v9x"
Jul 24 17:07:47.274540 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.274514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-468pm\" (UniqueName: \"kubernetes.io/projected/99ba6f44-50af-47c9-9fec-370390ebfd14-kube-api-access-468pm\") pod \"downloads-7c97dc4555-r2v9x\" (UID: \"99ba6f44-50af-47c9-9fec-370390ebfd14\") " pod="openshift-console/downloads-7c97dc4555-r2v9x"
Jul 24 17:07:47.466990 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.466967    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-r2v9x"
Jul 24 17:07:47.590684 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.590629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-r2v9x"]
Jul 24 17:07:47.593704 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:07:47.593673    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ba6f44_50af_47c9_9fec_370390ebfd14.slice/crio-17ff440d416aeb0a032123edc41e021308ae0fbac2432e1f33971bcc2948df52 WatchSource:0}: Error finding container 17ff440d416aeb0a032123edc41e021308ae0fbac2432e1f33971bcc2948df52: Status 404 returned error can't find the container with id 17ff440d416aeb0a032123edc41e021308ae0fbac2432e1f33971bcc2948df52
Jul 24 17:07:47.615806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.615778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pb8t2" event={"ID":"726e53f0-e3d8-4414-929b-683f5f4eb5c2","Type":"ContainerStarted","Data":"40edf63636af096a6d703db9d7b876ebf96de7d968a3d0e4065863bacd26c26e"}
Jul 24 17:07:47.616779 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.616757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-r2v9x" event={"ID":"99ba6f44-50af-47c9-9fec-370390ebfd14","Type":"ContainerStarted","Data":"17ff440d416aeb0a032123edc41e021308ae0fbac2432e1f33971bcc2948df52"}
Jul 24 17:07:47.631952 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:47.631905    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pb8t2" podStartSLOduration=16.7622817 podStartE2EDuration="28.631893031s" totalImagesPullingTime="11.869611331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:07:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:07:47.631254484 +0000 UTC m=+271.462507170" watchObservedRunningTime="2026-07-24 17:07:47.631893031 +0000 UTC m=+271.463145718"
Jul 24 17:07:49.923749 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:49.923712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:50.021169 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.021135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:50.087724 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.087695    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"]
Jul 24 17:07:50.090705 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.090679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"
Jul 24 17:07:50.092864 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.092837    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 24 17:07:50.093141 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.093117    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-btj92\""
Jul 24 17:07:50.099703 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.099679    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"]
Jul 24 17:07:50.105238 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.105214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:50.182008 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.181930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/42611f4e-4bca-4b5e-b6f0-e68314d12922-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2gfmq\" (UID: \"42611f4e-4bca-4b5e-b6f0-e68314d12922\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"
Jul 24 17:07:50.282859 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.282818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/42611f4e-4bca-4b5e-b6f0-e68314d12922-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2gfmq\" (UID: \"42611f4e-4bca-4b5e-b6f0-e68314d12922\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"
Jul 24 17:07:50.285846 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.285820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/42611f4e-4bca-4b5e-b6f0-e68314d12922-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2gfmq\" (UID: \"42611f4e-4bca-4b5e-b6f0-e68314d12922\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"
Jul 24 17:07:50.307133 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.307107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:50.402409 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.402380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"
Jul 24 17:07:50.540687 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.540635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"]
Jul 24 17:07:50.547264 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:07:50.547226    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42611f4e_4bca_4b5e_b6f0_e68314d12922.slice/crio-19a69807fd33ce27d009f20959d9d21b603b408fce5c1b092d2ceb767a7c0180 WatchSource:0}: Error finding container 19a69807fd33ce27d009f20959d9d21b603b408fce5c1b092d2ceb767a7c0180: Status 404 returned error can't find the container with id 19a69807fd33ce27d009f20959d9d21b603b408fce5c1b092d2ceb767a7c0180
Jul 24 17:07:50.627349 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:50.627318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq" event={"ID":"42611f4e-4bca-4b5e-b6f0-e68314d12922","Type":"ContainerStarted","Data":"19a69807fd33ce27d009f20959d9d21b603b408fce5c1b092d2ceb767a7c0180"}
Jul 24 17:07:52.182490 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:52.182383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:52.283189 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:52.283139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:52.637210 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:52.637173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq" event={"ID":"42611f4e-4bca-4b5e-b6f0-e68314d12922","Type":"ContainerStarted","Data":"e2d42b475480a0bd808dafb855ed5abd4b78ee8d87017f41c125c579a0be7cb0"}
Jul 24 17:07:52.637425 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:52.637405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"
Jul 24 17:07:52.642780 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:52.642753    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq"
Jul 24 17:07:52.653694 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:52.653631    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2gfmq" podStartSLOduration=1.483109329 podStartE2EDuration="2.653613335s" totalImagesPullingTime="1.170504006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:07:52.652724447 +0000 UTC m=+276.483977135" watchObservedRunningTime="2026-07-24 17:07:52.653613335 +0000 UTC m=+276.484866023"
Jul 24 17:07:53.174468 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.174433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:53.218760 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.218733    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"]
Jul 24 17:07:53.222087 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.222066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.224491 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.224465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 17:07:53.224615 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.224591    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 17:07:53.225201 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.225181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 17:07:53.225295 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.225181    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-njzw6\""
Jul 24 17:07:53.225295 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.225227    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 17:07:53.225397 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.225316    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 17:07:53.233382 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.233362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"]
Jul 24 17:07:53.256599 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.256574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:53.302849 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.302808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6sn\" (UniqueName: \"kubernetes.io/projected/bf21d174-fa96-4af3-b381-317d9415785d-kube-api-access-qh6sn\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.302969 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.302875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf21d174-fa96-4af3-b381-317d9415785d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.302969 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.302953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf21d174-fa96-4af3-b381-317d9415785d-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.303090 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.302995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf21d174-fa96-4af3-b381-317d9415785d-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.346840 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.346815    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:53.404015 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.403993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf21d174-fa96-4af3-b381-317d9415785d-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.404134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.404028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf21d174-fa96-4af3-b381-317d9415785d-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.404134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.404062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6sn\" (UniqueName: \"kubernetes.io/projected/bf21d174-fa96-4af3-b381-317d9415785d-kube-api-access-qh6sn\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.404134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.404116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf21d174-fa96-4af3-b381-317d9415785d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.404756 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.404733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf21d174-fa96-4af3-b381-317d9415785d-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.406874 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.406831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf21d174-fa96-4af3-b381-317d9415785d-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.406874 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.406850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf21d174-fa96-4af3-b381-317d9415785d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.415596 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.415579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6sn\" (UniqueName: \"kubernetes.io/projected/bf21d174-fa96-4af3-b381-317d9415785d-kube-api-access-qh6sn\") pod \"prometheus-operator-84d69f794d-d9cjn\" (UID: \"bf21d174-fa96-4af3-b381-317d9415785d\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.532910 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.532883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"
Jul 24 17:07:53.677879 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:53.677847    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-d9cjn"]
Jul 24 17:07:53.682478 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:07:53.682452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf21d174_fa96_4af3_b381_317d9415785d.slice/crio-481a75690cf42f3fa93ceaa97626005deb3498adbe5d78cc5106a5452aa1355b WatchSource:0}: Error finding container 481a75690cf42f3fa93ceaa97626005deb3498adbe5d78cc5106a5452aa1355b: Status 404 returned error can't find the container with id 481a75690cf42f3fa93ceaa97626005deb3498adbe5d78cc5106a5452aa1355b
Jul 24 17:07:54.567560 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:54.567512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:54.647437 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:54.647404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn" event={"ID":"bf21d174-fa96-4af3-b381-317d9415785d","Type":"ContainerStarted","Data":"481a75690cf42f3fa93ceaa97626005deb3498adbe5d78cc5106a5452aa1355b"}
Jul 24 17:07:54.668707 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:54.668674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:55.351403 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.351296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:55.532889 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.532863    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54b778d658-khvzj"]
Jul 24 17:07:55.536005 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.535983    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.539212 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.539087    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hjbgn\""
Jul 24 17:07:55.539212 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.539173    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 17:07:55.539212 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.539173    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 17:07:55.539481 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.539229    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 17:07:55.539481 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.539463    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 17:07:55.539611 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.539508    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 17:07:55.544035 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.544008    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 17:07:55.547188 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.547170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54b778d658-khvzj"]
Jul 24 17:07:55.622445 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.622362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-oauth-serving-cert\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.622445 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.622400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-oauth-config\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.622445 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.622425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-console-config\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.622900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.622465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbcp7\" (UniqueName: \"kubernetes.io/projected/1419a624-0643-449d-90d4-95792935437e-kube-api-access-dbcp7\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.622900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.622494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-serving-cert\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.622900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.622540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-service-ca\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.622900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.622595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-trusted-ca-bundle\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.652376 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.652344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn" event={"ID":"bf21d174-fa96-4af3-b381-317d9415785d","Type":"ContainerStarted","Data":"10c201b5ddcfb609478f105f5f1986498929e3eb80c465c0a58ecdaa2838162e"}
Jul 24 17:07:55.652503 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.652396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn" event={"ID":"bf21d174-fa96-4af3-b381-317d9415785d","Type":"ContainerStarted","Data":"7a20166aa0e40557cf947af29192c5442853339e1480e7b7d54071a510449980"}
Jul 24 17:07:55.670901 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.670846    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-d9cjn" podStartSLOduration=1.346579471 podStartE2EDuration="2.670834508s" totalImagesPullingTime="1.324255037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:07:55.670467849 +0000 UTC m=+279.501720540" watchObservedRunningTime="2026-07-24 17:07:55.670834508 +0000 UTC m=+279.502087196"
Jul 24 17:07:55.723557 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.723529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-oauth-config\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.723713 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.723566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-console-config\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.723798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.723769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcp7\" (UniqueName: \"kubernetes.io/projected/1419a624-0643-449d-90d4-95792935437e-kube-api-access-dbcp7\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.723871 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.723821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-serving-cert\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.723871 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.723843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-service-ca\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.723976 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.723919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-trusted-ca-bundle\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.724028 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.723993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-oauth-serving-cert\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.724544 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.724334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-console-config\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.724637 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.724600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-oauth-serving-cert\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.724787 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.724765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-service-ca\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.725145 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.725112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-trusted-ca-bundle\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.727062 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.727040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-oauth-config\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.727142 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.727084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-serving-cert\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.732484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.732457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbcp7\" (UniqueName: \"kubernetes.io/projected/1419a624-0643-449d-90d4-95792935437e-kube-api-access-dbcp7\") pod \"console-54b778d658-khvzj\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") " pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.847532 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.847503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:07:55.991940 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:55.991909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54b778d658-khvzj"]
Jul 24 17:07:55.997634 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:07:55.997607    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1419a624_0643_449d_90d4_95792935437e.slice/crio-6095dd1c20b4cc9092d8f9ce394e907ecd4da395c84d48688ef2658a67a42f6a WatchSource:0}: Error finding container 6095dd1c20b4cc9092d8f9ce394e907ecd4da395c84d48688ef2658a67a42f6a: Status 404 returned error can't find the container with id 6095dd1c20b4cc9092d8f9ce394e907ecd4da395c84d48688ef2658a67a42f6a
Jul 24 17:07:56.658529 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:56.658480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b778d658-khvzj" event={"ID":"1419a624-0643-449d-90d4-95792935437e","Type":"ContainerStarted","Data":"6095dd1c20b4cc9092d8f9ce394e907ecd4da395c84d48688ef2658a67a42f6a"}
Jul 24 17:07:57.313688 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.313636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:57.601934 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.601859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:57.613056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.613029    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"]
Jul 24 17:07:57.617598 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.617576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.620570 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.620400    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4zjjg\""
Jul 24 17:07:57.620570 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.620469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 24 17:07:57.620761 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.620610    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 24 17:07:57.632553 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.632531    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"]
Jul 24 17:07:57.637692 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.637672    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6ns2w"]
Jul 24 17:07:57.640177 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.640150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff207798-7666-45d2-84fa-3737f4e6ad65-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.640274 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.640222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff207798-7666-45d2-84fa-3737f4e6ad65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.640274 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.640261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff207798-7666-45d2-84fa-3737f4e6ad65-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.640376 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.640305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xk6d\" (UniqueName: \"kubernetes.io/projected/ff207798-7666-45d2-84fa-3737f4e6ad65-kube-api-access-7xk6d\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.641132 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.641106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.643416 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.643397    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 17:07:57.643558 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.643542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 17:07:57.643630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.643600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 17:07:57.643731 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.643715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dktwt\""
Jul 24 17:07:57.690270 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.690232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:57.740677 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff207798-7666-45d2-84fa-3737f4e6ad65-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.740794 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-wtmp\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.740794 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b62d8c1-cb04-4645-9684-793f2a261d22-metrics-client-ca\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.740794 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.740794 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.741056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-textfile\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.741056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff207798-7666-45d2-84fa-3737f4e6ad65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.741056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff207798-7666-45d2-84fa-3737f4e6ad65-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.741056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b62d8c1-cb04-4645-9684-793f2a261d22-sys\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.741056 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:57.740956    2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 24 17:07:57.741056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.740986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xk6d\" (UniqueName: \"kubernetes.io/projected/ff207798-7666-45d2-84fa-3737f4e6ad65-kube-api-access-7xk6d\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.741056 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:57.741014    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff207798-7666-45d2-84fa-3737f4e6ad65-openshift-state-metrics-tls podName:ff207798-7666-45d2-84fa-3737f4e6ad65 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:58.240996061 +0000 UTC m=+282.072248727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ff207798-7666-45d2-84fa-3737f4e6ad65-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-7lz8q" (UID: "ff207798-7666-45d2-84fa-3737f4e6ad65") : secret "openshift-state-metrics-tls" not found
Jul 24 17:07:57.741056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.741037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b62d8c1-cb04-4645-9684-793f2a261d22-root\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.741388 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.741072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-tls\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.741388 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.741104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kkl\" (UniqueName: \"kubernetes.io/projected/8b62d8c1-cb04-4645-9684-793f2a261d22-kube-api-access-25kkl\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.741724 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.741707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff207798-7666-45d2-84fa-3737f4e6ad65-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.743538 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.743516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff207798-7666-45d2-84fa-3737f4e6ad65-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.749426 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.749403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xk6d\" (UniqueName: \"kubernetes.io/projected/ff207798-7666-45d2-84fa-3737f4e6ad65-kube-api-access-7xk6d\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:57.760882 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.760860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:57.842289 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-wtmp\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842415 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b62d8c1-cb04-4645-9684-793f2a261d22-metrics-client-ca\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842474 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842474 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-wtmp\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842621 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842621 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-textfile\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842621 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b62d8c1-cb04-4645-9684-793f2a261d22-sys\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b62d8c1-cb04-4645-9684-793f2a261d22-sys\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b62d8c1-cb04-4645-9684-793f2a261d22-root\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-tls\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842950 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kkl\" (UniqueName: \"kubernetes.io/projected/8b62d8c1-cb04-4645-9684-793f2a261d22-kube-api-access-25kkl\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842950 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b62d8c1-cb04-4645-9684-793f2a261d22-root\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842950 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.842897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b62d8c1-cb04-4645-9684-793f2a261d22-metrics-client-ca\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.842950 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:57.842927    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 17:07:57.843136 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:57.842981    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-tls podName:8b62d8c1-cb04-4645-9684-793f2a261d22 nodeName:}" failed. No retries permitted until 2026-07-24 17:07:58.342965071 +0000 UTC m=+282.174217736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-tls") pod "node-exporter-6ns2w" (UID: "8b62d8c1-cb04-4645-9684-793f2a261d22") : secret "node-exporter-tls" not found
Jul 24 17:07:57.843136 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.843051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.843136 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.843124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-textfile\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.845142 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.845123    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.851017 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.850975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kkl\" (UniqueName: \"kubernetes.io/projected/8b62d8c1-cb04-4645-9684-793f2a261d22-kube-api-access-25kkl\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:57.860750 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.860674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:57.963340 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:57.963303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:58.065969 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.065931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:58.246284 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.246255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff207798-7666-45d2-84fa-3737f4e6ad65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:58.248969 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.248946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff207798-7666-45d2-84fa-3737f4e6ad65-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-7lz8q\" (UID: \"ff207798-7666-45d2-84fa-3737f4e6ad65\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:58.347565 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.347537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-tls\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:58.350169 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.350141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b62d8c1-cb04-4645-9684-793f2a261d22-node-exporter-tls\") pod \"node-exporter-6ns2w\" (UID: \"8b62d8c1-cb04-4645-9684-793f2a261d22\") " pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:58.371089 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.371057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:58.529691 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.529588    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"
Jul 24 17:07:58.552535 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.552511    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ns2w"
Jul 24 17:07:58.679509 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.679465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:58.766541 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.766505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:58.771025 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.771005    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:07:58.774987 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.774878    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.777815 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.777796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 17:07:58.778184 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.778162    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 17:07:58.778609 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.778588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 17:07:58.778765 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.778608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 17:07:58.778765 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.778674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 17:07:58.778765 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.778718    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v2qd7\""
Jul 24 17:07:58.778978 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.778867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 17:07:58.778978 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.778924    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 17:07:58.778978 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.778937    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 17:07:58.794189 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.794170    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 17:07:58.803323 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.803302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:07:58.851788 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.851757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.851924 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.851811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-web-config\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.851924 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.851839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e40b7a7-a81d-4283-857b-253a273d952a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.851924 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.851863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.851924 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.851901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.852119 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.851940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e40b7a7-a81d-4283-857b-253a273d952a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.852119 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.851956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e40b7a7-a81d-4283-857b-253a273d952a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.852119 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.851980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.852119 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.852064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849kk\" (UniqueName: \"kubernetes.io/projected/5e40b7a7-a81d-4283-857b-253a273d952a-kube-api-access-849kk\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.852276 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.852138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.852276 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.852213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e40b7a7-a81d-4283-857b-253a273d952a-config-out\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.852276 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.852235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e40b7a7-a81d-4283-857b-253a273d952a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.852276 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.852260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.861682 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.861629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:58.953220 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e40b7a7-a81d-4283-857b-253a273d952a-config-out\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953393 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e40b7a7-a81d-4283-857b-253a273d952a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953393 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953393 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-web-config\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e40b7a7-a81d-4283-857b-253a273d952a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953782 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e40b7a7-a81d-4283-857b-253a273d952a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953782 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e40b7a7-a81d-4283-857b-253a273d952a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953782 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953782 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849kk\" (UniqueName: \"kubernetes.io/projected/5e40b7a7-a81d-4283-857b-253a273d952a-kube-api-access-849kk\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.953782 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.954006 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.953877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e40b7a7-a81d-4283-857b-253a273d952a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.954061 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:58.954018    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e40b7a7-a81d-4283-857b-253a273d952a-alertmanager-trusted-ca-bundle podName:5e40b7a7-a81d-4283-857b-253a273d952a nodeName:}" failed. No retries permitted until 2026-07-24 17:07:59.453997755 +0000 UTC m=+283.285250420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e40b7a7-a81d-4283-857b-253a273d952a-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5e40b7a7-a81d-4283-857b-253a273d952a") : configmap references non-existent config key: ca-bundle.crt
Jul 24 17:07:58.955418 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.955338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e40b7a7-a81d-4283-857b-253a273d952a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.956483 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.956359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e40b7a7-a81d-4283-857b-253a273d952a-config-out\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.957331 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.957306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.957600 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.957407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.957600 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.957463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.957806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.957782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e40b7a7-a81d-4283-857b-253a273d952a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.958461 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.958431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.958591 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.958573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.959459 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.959436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.959558 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.959516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e40b7a7-a81d-4283-857b-253a273d952a-web-config\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:58.964178 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:58.964154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-849kk\" (UniqueName: \"kubernetes.io/projected/5e40b7a7-a81d-4283-857b-253a273d952a-kube-api-access-849kk\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:59.309135 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:07:59.309090    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zk69n" podUID="d8201c68-e268-411e-886e-87a4c1429052"
Jul 24 17:07:59.458585 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:59.458544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e40b7a7-a81d-4283-857b-253a273d952a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:59.459455 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:59.459427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e40b7a7-a81d-4283-857b-253a273d952a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e40b7a7-a81d-4283-857b-253a273d952a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:07:59.674398 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:59.674315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zk69n"
Jul 24 17:07:59.686386 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:07:59.686359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:08:00.467003 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:00.466956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:00.521396 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:00.521343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:00.690380 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:00.690345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:00.825452 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:00.825371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:02.132319 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:02.132286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:02.191414 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:02.191383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:02.327724 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:02.327693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:02.399899 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:02.399801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:02.586793 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:02.586754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:08:02.589425 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:02.589395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8201c68-e268-411e-886e-87a4c1429052-metrics-tls\") pod \"dns-default-zk69n\" (UID: \"d8201c68-e268-411e-886e-87a4c1429052\") " pod="openshift-dns/dns-default-zk69n"
Jul 24 17:08:02.601253 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:02.601197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:02.677708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:02.677617    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-57f2l\""
Jul 24 17:08:02.685518 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:02.685492    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zk69n"
Jul 24 17:08:03.449026 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.448993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:03.549944 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.549908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:03.612047 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.612010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:03.742465 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.742432    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:08:03.744847 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.744814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:03.754408 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.754387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.756871 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.756841    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 17:08:03.757755 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.757709    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 17:08:03.757911 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.757787    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mv8lc\""
Jul 24 17:08:03.757911 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.757896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 17:08:03.758044 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.757918    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 17:08:03.758044 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.757929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 17:08:03.758274 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.758253    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 17:08:03.758435 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.758361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 17:08:03.758435 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.758374    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 17:08:03.758435 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.758395    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 17:08:03.758621 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.758443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 17:08:03.758621 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.758526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 17:08:03.759165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.758985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-48sdg21piba1b\""
Jul 24 17:08:03.759165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.759063    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 17:08:03.759165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.759115    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 17:08:03.759165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.759164    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 17:08:03.762167 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.762146    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 17:08:03.765693 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.765385    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:08:03.799266 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799446 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799446 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799446 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799446 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799720 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799720 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8889927-f4b8-4b6b-93f6-50fe15fef290-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799720 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799720 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799720 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799720 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799720 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.799720 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-config\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.800134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8889927-f4b8-4b6b-93f6-50fe15fef290-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.800134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.800134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8vh\" (UniqueName: \"kubernetes.io/projected/b8889927-f4b8-4b6b-93f6-50fe15fef290-kube-api-access-ds8vh\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.800134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.800134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.800134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.800134 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.799953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.900477 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.900691 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.900691 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.900691 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.900691 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.900691 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.900691 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8889927-f4b8-4b6b-93f6-50fe15fef290-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-config\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8889927-f4b8-4b6b-93f6-50fe15fef290-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8vh\" (UniqueName: \"kubernetes.io/projected/b8889927-f4b8-4b6b-93f6-50fe15fef290-kube-api-access-ds8vh\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.900978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.901000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901577 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.901036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901577 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.901063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901577 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.901498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.901736 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.901589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.902089 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.902063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.902207 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.902137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.902207 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.902180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.905131 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.902426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.905131 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.903139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.905462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.905416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8889927-f4b8-4b6b-93f6-50fe15fef290-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.906048 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.906005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8889927-f4b8-4b6b-93f6-50fe15fef290-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.906757 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.906677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8889927-f4b8-4b6b-93f6-50fe15fef290-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.907076 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.907051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.907160 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.907103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.907669 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.907625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-config\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.907745 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.907637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.908201 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.908147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.909059 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.909038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.909281 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.909259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.909389 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.909370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.909454 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.909370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8889927-f4b8-4b6b-93f6-50fe15fef290-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:03.910812 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:03.910780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8vh\" (UniqueName: \"kubernetes.io/projected/b8889927-f4b8-4b6b-93f6-50fe15fef290-kube-api-access-ds8vh\") pod \"prometheus-k8s-0\" (UID: \"b8889927-f4b8-4b6b-93f6-50fe15fef290\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:04.071507 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:04.071417    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:05.151687 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:05.151627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:06.192493 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:08:06.192264    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b62d8c1_cb04_4645_9684_793f2a261d22.slice/crio-560d221b8313ba52364371177c17fec0e8420e563e6991233c4438ca4a05136f WatchSource:0}: Error finding container 560d221b8313ba52364371177c17fec0e8420e563e6991233c4438ca4a05136f: Status 404 returned error can't find the container with id 560d221b8313ba52364371177c17fec0e8420e563e6991233c4438ca4a05136f
Jul 24 17:08:06.288454 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.288362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:06.388125 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.387998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:08:06.401747 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:08:06.400681    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e40b7a7_a81d_4283_857b_253a273d952a.slice/crio-6a86e9e69d81ee8a22331d8505fc7668e46bad5a9e6dae3b15b77565063dd88b WatchSource:0}: Error finding container 6a86e9e69d81ee8a22331d8505fc7668e46bad5a9e6dae3b15b77565063dd88b: Status 404 returned error can't find the container with id 6a86e9e69d81ee8a22331d8505fc7668e46bad5a9e6dae3b15b77565063dd88b
Jul 24 17:08:06.406532 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.406508    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zk69n"]
Jul 24 17:08:06.414034 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:08:06.413994    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8201c68_e268_411e_886e_87a4c1429052.slice/crio-cfa2e970666fe1a94cd3075525c9b97cf5c0546657970feb5d222db3204f3ca3 WatchSource:0}: Error finding container cfa2e970666fe1a94cd3075525c9b97cf5c0546657970feb5d222db3204f3ca3: Status 404 returned error can't find the container with id cfa2e970666fe1a94cd3075525c9b97cf5c0546657970feb5d222db3204f3ca3
Jul 24 17:08:06.424244 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.424217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q"]
Jul 24 17:08:06.427621 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:08:06.427566    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff207798_7666_45d2_84fa_3737f4e6ad65.slice/crio-1d207c5be906c4d57649c2f16d1349f00aad1813f90ffcb0e87cf3b08fc1781e WatchSource:0}: Error finding container 1d207c5be906c4d57649c2f16d1349f00aad1813f90ffcb0e87cf3b08fc1781e: Status 404 returned error can't find the container with id 1d207c5be906c4d57649c2f16d1349f00aad1813f90ffcb0e87cf3b08fc1781e
Jul 24 17:08:06.456858 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.456829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:08:06.469715 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:08:06.469689    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8889927_f4b8_4b6b_93f6_50fe15fef290.slice/crio-b081eef712ab165517d011f67884ceaf527aa7baaa91c49526c9f044f7bd629e WatchSource:0}: Error finding container b081eef712ab165517d011f67884ceaf527aa7baaa91c49526c9f044f7bd629e: Status 404 returned error can't find the container with id b081eef712ab165517d011f67884ceaf527aa7baaa91c49526c9f044f7bd629e
Jul 24 17:08:06.699404 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.699317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zk69n" event={"ID":"d8201c68-e268-411e-886e-87a4c1429052","Type":"ContainerStarted","Data":"cfa2e970666fe1a94cd3075525c9b97cf5c0546657970feb5d222db3204f3ca3"}
Jul 24 17:08:06.702868 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.702839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8889927-f4b8-4b6b-93f6-50fe15fef290","Type":"ContainerStarted","Data":"b081eef712ab165517d011f67884ceaf527aa7baaa91c49526c9f044f7bd629e"}
Jul 24 17:08:06.704636 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.704604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ns2w" event={"ID":"8b62d8c1-cb04-4645-9684-793f2a261d22","Type":"ContainerStarted","Data":"560d221b8313ba52364371177c17fec0e8420e563e6991233c4438ca4a05136f"}
Jul 24 17:08:06.707132 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.706709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-r2v9x" event={"ID":"99ba6f44-50af-47c9-9fec-370390ebfd14","Type":"ContainerStarted","Data":"629199d11a892bef2419f7abbead399551568252c52f89ffbbfa1218dfb9315b"}
Jul 24 17:08:06.707132 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.706934    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-r2v9x"
Jul 24 17:08:06.709684 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.709662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q" event={"ID":"ff207798-7666-45d2-84fa-3737f4e6ad65","Type":"ContainerStarted","Data":"bc671dbe155926399d353d714d3ac9edf14e8d7375cd6c9c57396d637c701334"}
Jul 24 17:08:06.709841 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.709706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q" event={"ID":"ff207798-7666-45d2-84fa-3737f4e6ad65","Type":"ContainerStarted","Data":"c118b6640de73a567a9739ca293660d961ea0c297f601999c06b26d92ebaf98c"}
Jul 24 17:08:06.709841 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.709737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q" event={"ID":"ff207798-7666-45d2-84fa-3737f4e6ad65","Type":"ContainerStarted","Data":"1d207c5be906c4d57649c2f16d1349f00aad1813f90ffcb0e87cf3b08fc1781e"}
Jul 24 17:08:06.711875 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.711740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b778d658-khvzj" event={"ID":"1419a624-0643-449d-90d4-95792935437e","Type":"ContainerStarted","Data":"f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5"}
Jul 24 17:08:06.713859 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.713773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e40b7a7-a81d-4283-857b-253a273d952a","Type":"ContainerStarted","Data":"6a86e9e69d81ee8a22331d8505fc7668e46bad5a9e6dae3b15b77565063dd88b"}
Jul 24 17:08:06.719183 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.719161    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-r2v9x"
Jul 24 17:08:06.724581 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:06.724530    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-r2v9x" podStartSLOduration=1.109228553 podStartE2EDuration="19.724513144s" totalImagesPullingTime="18.615284591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:06.722955043 +0000 UTC m=+290.554207731" watchObservedRunningTime="2026-07-24 17:08:06.724513144 +0000 UTC m=+290.555765833"
Jul 24 17:08:07.352850 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:07.352810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:07.411242 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:07.411208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:08.950571 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:08.950523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:09.050075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:09.050034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:10.258538 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.258508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:10.314449 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.314407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:10.732855 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.732825    2563 generic.go:356] "Generic (PLEG): container finished" podID="5e40b7a7-a81d-4283-857b-253a273d952a" containerID="0d4797b9759529affa6a33b1b61178c97743fef571bac4d630c25db12ba006c2" exitCode=0
Jul 24 17:08:10.733100 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.732915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e40b7a7-a81d-4283-857b-253a273d952a","Type":"ContainerDied","Data":"0d4797b9759529affa6a33b1b61178c97743fef571bac4d630c25db12ba006c2"}
Jul 24 17:08:10.734982 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.734958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zk69n" event={"ID":"d8201c68-e268-411e-886e-87a4c1429052","Type":"ContainerStarted","Data":"75e95aa635aab35f1c20493ba57ce1bee5b7e0a79e113712bdb77ea6b71af439"}
Jul 24 17:08:10.735107 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.735007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zk69n" event={"ID":"d8201c68-e268-411e-886e-87a4c1429052","Type":"ContainerStarted","Data":"8990d9a883204c1538cb790c7db40079820d1990f825578d711863cae7dfd8e6"}
Jul 24 17:08:10.735107 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.735051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zk69n"
Jul 24 17:08:10.737064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.737037    2563 generic.go:356] "Generic (PLEG): container finished" podID="b8889927-f4b8-4b6b-93f6-50fe15fef290" containerID="8c6986fb8afed4754d563edd4ac09b904f754053ad491ccf0925cded8eb03ec8" exitCode=0
Jul 24 17:08:10.737161 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.737120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8889927-f4b8-4b6b-93f6-50fe15fef290","Type":"ContainerDied","Data":"8c6986fb8afed4754d563edd4ac09b904f754053ad491ccf0925cded8eb03ec8"}
Jul 24 17:08:10.739335 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.739315    2563 generic.go:356] "Generic (PLEG): container finished" podID="8b62d8c1-cb04-4645-9684-793f2a261d22" containerID="9d8e8724a6228081511a3b3eea0da336d48c1129c918adf8efbf97621999dbd7" exitCode=0
Jul 24 17:08:10.739431 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.739411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ns2w" event={"ID":"8b62d8c1-cb04-4645-9684-793f2a261d22","Type":"ContainerDied","Data":"9d8e8724a6228081511a3b3eea0da336d48c1129c918adf8efbf97621999dbd7"}
Jul 24 17:08:10.743500 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.743482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q" event={"ID":"ff207798-7666-45d2-84fa-3737f4e6ad65","Type":"ContainerStarted","Data":"18ff23128ee0ada9aebb0ba6d6f1a916968af4c9d111f8c50832173dfed89fee"}
Jul 24 17:08:10.760314 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.760274    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54b778d658-khvzj" podStartSLOduration=5.561616335 podStartE2EDuration="15.760257722s" totalImagesPullingTime="10.198641387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:06.757293953 +0000 UTC m=+290.588546642" watchObservedRunningTime="2026-07-24 17:08:10.760257722 +0000 UTC m=+294.591510410"
Jul 24 17:08:10.803611 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.803545    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zk69n" podStartSLOduration=255.189779857 podStartE2EDuration="4m18.803523562s" totalImagesPullingTime="3.613743705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:10.803118786 +0000 UTC m=+294.634371485" watchObservedRunningTime="2026-07-24 17:08:10.803523562 +0000 UTC m=+294.634776250"
Jul 24 17:08:10.822428 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.822379    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-7lz8q" podStartSLOduration=10.373576207 podStartE2EDuration="13.822360186s" totalImagesPullingTime="3.448783979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:10.821050547 +0000 UTC m=+294.652303231" watchObservedRunningTime="2026-07-24 17:08:10.822360186 +0000 UTC m=+294.653612874"
Jul 24 17:08:10.862820 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.862791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:10.962490 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:10.962463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:11.455998 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:11.455899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:11.554015 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:11.553981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:11.755620 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:11.755577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ns2w" event={"ID":"8b62d8c1-cb04-4645-9684-793f2a261d22","Type":"ContainerStarted","Data":"c46ffd5818ff1be5777db7ff27243321b507a23d7dc91701e3abeab0051b71f6"}
Jul 24 17:08:11.755830 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:11.755665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ns2w" event={"ID":"8b62d8c1-cb04-4645-9684-793f2a261d22","Type":"ContainerStarted","Data":"f35553606b67cbcb3f108bd92635b27b5392e007d462c3ebf8c415ce911f0c8b"}
Jul 24 17:08:13.143979 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:13.143451    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6ns2w" podStartSLOduration=12.208642935 podStartE2EDuration="16.143433538s" totalImagesPullingTime="3.82780245s" totalInitContainerRuntime="106.988153ms" isStatefulPod=true podCreationTimestamp="2026-07-24 17:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:11.778446058 +0000 UTC m=+295.609698747" watchObservedRunningTime="2026-07-24 17:08:13.143433538 +0000 UTC m=+296.974686232"
Jul 24 17:08:13.143979 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:13.143831    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54b778d658-khvzj"]
Jul 24 17:08:13.406898 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:13.406819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:13.505343 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:13.505243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:13.683818 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:08:13.683224    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-w8hqg" podUID="07768a9d-923d-4e34-9c7d-4db82a6cf6c2"
Jul 24 17:08:13.774808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:13.774756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e40b7a7-a81d-4283-857b-253a273d952a","Type":"ContainerStarted","Data":"199a1dd9cc3857637a8788fb8ea3686bb2648807f7c5ef1fbd83e3be233bd7ea"}
Jul 24 17:08:13.774997 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:13.774836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e40b7a7-a81d-4283-857b-253a273d952a","Type":"ContainerStarted","Data":"1cae563f8af3fc483f7ad5badcac251432a22966d5252160e60cde0a0aaff1dc"}
Jul 24 17:08:13.774997 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:13.774880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e40b7a7-a81d-4283-857b-253a273d952a","Type":"ContainerStarted","Data":"5a00355816c3c3d88d97a0747071686da5c6d64bd0f4ae0fb7c117d9725388b9"}
Jul 24 17:08:13.774997 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:13.774920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e40b7a7-a81d-4283-857b-253a273d952a","Type":"ContainerStarted","Data":"60468ff8f799d99275b66f1735d70bdea419cc0a48a42ad716d4d7c465d571d2"}
Jul 24 17:08:15.798146 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:15.798108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e40b7a7-a81d-4283-857b-253a273d952a","Type":"ContainerStarted","Data":"6082b3b74cd0674407ecf41813fd4c42ada1d2de07c577978389391dc56c01a9"}
Jul 24 17:08:15.800684 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:15.800629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8889927-f4b8-4b6b-93f6-50fe15fef290","Type":"ContainerStarted","Data":"a97a0db8671fcebd54c684c63d412c1c2b56f14d323ae73ba1cb4e0b0e7922bf"}
Jul 24 17:08:15.800829 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:15.800730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8889927-f4b8-4b6b-93f6-50fe15fef290","Type":"ContainerStarted","Data":"df2ba327917846f87b71b2ca07af9e3315be41f3f9c82f44c34d2e300ee4e787"}
Jul 24 17:08:15.848363 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:15.848331    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:08:16.549682 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:16.548920    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:08:16.550746 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:16.550628    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:08:16.558438 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:16.558347    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 17:08:16.685791 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:16.685619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:17.097277 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:16.771948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:17.097277 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:16.810982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e40b7a7-a81d-4283-857b-253a273d952a","Type":"ContainerStarted","Data":"3fb0089ef07f2573888db1ac861a55d8aa50cf20d2b75e0de650d81180ae672f"}
Jul 24 17:08:17.097277 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:16.835267    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=11.512679367 podStartE2EDuration="18.835246969s" totalImagesPullingTime="7.300566517s" totalInitContainerRuntime="22.001085ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:07:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:16.83420988 +0000 UTC m=+300.665462566" watchObservedRunningTime="2026-07-24 17:08:16.835246969 +0000 UTC m=+300.666499657"
Jul 24 17:08:17.551299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:17.551260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:17.612958 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:17.612927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:18.564513 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:18.564479    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" podUID="0b5e99fb-8044-4e81-a95e-295982f1fa88" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 24 17:08:18.824761 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:18.824720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8889927-f4b8-4b6b-93f6-50fe15fef290","Type":"ContainerStarted","Data":"240e390e6e4490b38d5171e712c3a7c0a26de682afbfaa3ab1fa1b2e9c91c029"}
Jul 24 17:08:18.824912 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:18.824817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8889927-f4b8-4b6b-93f6-50fe15fef290","Type":"ContainerStarted","Data":"c0f29bc9094d2d80bcd76e4288e1dce05c5a3fcf758306fb88993537a058fb84"}
Jul 24 17:08:18.824912 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:18.824860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8889927-f4b8-4b6b-93f6-50fe15fef290","Type":"ContainerStarted","Data":"def3d698aeb9ef0c2624b19c9eafdcea571996e687a0b472b75688b46547fe14"}
Jul 24 17:08:19.831233 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:19.831179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8889927-f4b8-4b6b-93f6-50fe15fef290","Type":"ContainerStarted","Data":"22eee4fbfe371034247dc2775ff7ccfd33b020789563c8538179f326fa80c8df"}
Jul 24 17:08:19.859333 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:19.859285    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.893795556 podStartE2EDuration="16.859269293s" totalImagesPullingTime="10.961226148s" totalInitContainerRuntime="4.247589ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:08:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:19.857893348 +0000 UTC m=+303.689146035" watchObservedRunningTime="2026-07-24 17:08:19.859269293 +0000 UTC m=+303.690521980"
Jul 24 17:08:20.758442 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:20.758410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zk69n"
Jul 24 17:08:22.027430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:22.027396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:22.127777 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:22.127751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:24.071590 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:24.071557    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:08:24.984310 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:24.984275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:25.682224 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:25.682193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:08:25.774983 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:25.774943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:25.864365 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:25.864336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:28.564864 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:28.564824    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" podUID="0b5e99fb-8044-4e81-a95e-295982f1fa88" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 24 17:08:34.085101 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:34.085067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:34.204888 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:34.204858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:37.028891 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:37.028849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:37.102719 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:37.102671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:37.162909 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:37.162878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:38.181178 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.181125    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54b778d658-khvzj" podUID="1419a624-0643-449d-90d4-95792935437e" containerName="console" containerID="cri-o://f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5" gracePeriod=15
Jul 24 17:08:38.447429 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.447088    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:08:38.564637 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.564592    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" podUID="0b5e99fb-8044-4e81-a95e-295982f1fa88" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 24 17:08:38.564826 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.564704    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8"
Jul 24 17:08:38.565444 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.565402    2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c133acf0ca3cbdc50dfd12f2ac5a7087a68c0d24b4ee805f89a989acf6461d71"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 24 17:08:38.565528 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.565473    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" podUID="0b5e99fb-8044-4e81-a95e-295982f1fa88" containerName="service-proxy" containerID="cri-o://c133acf0ca3cbdc50dfd12f2ac5a7087a68c0d24b4ee805f89a989acf6461d71" gracePeriod=30
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.605272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbcp7\" (UniqueName: \"kubernetes.io/projected/1419a624-0643-449d-90d4-95792935437e-kube-api-access-dbcp7\") pod \"1419a624-0643-449d-90d4-95792935437e\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") "
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.605308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-serving-cert\") pod \"1419a624-0643-449d-90d4-95792935437e\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") "
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.605336    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-oauth-config\") pod \"1419a624-0643-449d-90d4-95792935437e\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") "
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.605404    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-console-config\") pod \"1419a624-0643-449d-90d4-95792935437e\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") "
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.605433    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-trusted-ca-bundle\") pod \"1419a624-0643-449d-90d4-95792935437e\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") "
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.605459    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-service-ca\") pod \"1419a624-0643-449d-90d4-95792935437e\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") "
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.605505    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-oauth-serving-cert\") pod \"1419a624-0643-449d-90d4-95792935437e\" (UID: \"1419a624-0643-449d-90d4-95792935437e\") "
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.605896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-console-config" (OuterVolumeSpecName: "console-config") pod "1419a624-0643-449d-90d4-95792935437e" (UID: "1419a624-0643-449d-90d4-95792935437e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.606488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1419a624-0643-449d-90d4-95792935437e" (UID: "1419a624-0643-449d-90d4-95792935437e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.606756    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-service-ca" (OuterVolumeSpecName: "service-ca") pod "1419a624-0643-449d-90d4-95792935437e" (UID: "1419a624-0643-449d-90d4-95792935437e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.606967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1419a624-0643-449d-90d4-95792935437e" (UID: "1419a624-0643-449d-90d4-95792935437e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:08:38.608261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.608199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1419a624-0643-449d-90d4-95792935437e-kube-api-access-dbcp7" (OuterVolumeSpecName: "kube-api-access-dbcp7") pod "1419a624-0643-449d-90d4-95792935437e" (UID: "1419a624-0643-449d-90d4-95792935437e"). InnerVolumeSpecName "kube-api-access-dbcp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:08:38.609080 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.609039    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1419a624-0643-449d-90d4-95792935437e" (UID: "1419a624-0643-449d-90d4-95792935437e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:08:38.611529 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.611492    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1419a624-0643-449d-90d4-95792935437e" (UID: "1419a624-0643-449d-90d4-95792935437e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:08:38.707351 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.707230    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-oauth-serving-cert\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:08:38.707351 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.707262    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbcp7\" (UniqueName: \"kubernetes.io/projected/1419a624-0643-449d-90d4-95792935437e-kube-api-access-dbcp7\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:08:38.707351 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.707276    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-serving-cert\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:08:38.707351 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.707290    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1419a624-0643-449d-90d4-95792935437e-console-oauth-config\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:08:38.707351 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.707304    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-console-config\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:08:38.707351 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.707316    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-trusted-ca-bundle\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:08:38.707351 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.707329    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1419a624-0643-449d-90d4-95792935437e-service-ca\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:08:38.901968 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.901937    2563 generic.go:356] "Generic (PLEG): container finished" podID="1419a624-0643-449d-90d4-95792935437e" containerID="f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5" exitCode=0
Jul 24 17:08:38.902143 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.902003    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b778d658-khvzj"
Jul 24 17:08:38.902143 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.902027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b778d658-khvzj" event={"ID":"1419a624-0643-449d-90d4-95792935437e","Type":"ContainerDied","Data":"f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5"}
Jul 24 17:08:38.902143 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.902078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b778d658-khvzj" event={"ID":"1419a624-0643-449d-90d4-95792935437e","Type":"ContainerDied","Data":"6095dd1c20b4cc9092d8f9ce394e907ecd4da395c84d48688ef2658a67a42f6a"}
Jul 24 17:08:38.902143 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.902116    2563 scope.go:117] "RemoveContainer" containerID="f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5"
Jul 24 17:08:38.905916 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.905891    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b5e99fb-8044-4e81-a95e-295982f1fa88" containerID="c133acf0ca3cbdc50dfd12f2ac5a7087a68c0d24b4ee805f89a989acf6461d71" exitCode=2
Jul 24 17:08:38.906029 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.905937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" event={"ID":"0b5e99fb-8044-4e81-a95e-295982f1fa88","Type":"ContainerDied","Data":"c133acf0ca3cbdc50dfd12f2ac5a7087a68c0d24b4ee805f89a989acf6461d71"}
Jul 24 17:08:38.906029 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.905980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4c6f6bbc-rshk8" event={"ID":"0b5e99fb-8044-4e81-a95e-295982f1fa88","Type":"ContainerStarted","Data":"9ad244d5126491491e9c6ed0d6de5f9a31abda7849841f5bbdff2dc5a32ee8e2"}
Jul 24 17:08:38.912662 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.912616    2563 scope.go:117] "RemoveContainer" containerID="f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5"
Jul 24 17:08:38.919216 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.919188    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54b778d658-khvzj"]
Jul 24 17:08:38.919364 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:08:38.919293    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5\": container with ID starting with f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5 not found: ID does not exist" containerID="f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5"
Jul 24 17:08:38.919364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.919325    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5"} err="failed to get container status \"f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5\": rpc error: code = NotFound desc = could not find container \"f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5\": container with ID starting with f65b7af304c6bf0becc7ef39180c795358f5a7318779331985ad2a38590817e5 not found: ID does not exist"
Jul 24 17:08:38.923169 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:38.923125    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54b778d658-khvzj"]
Jul 24 17:08:40.687264 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:40.687231    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1419a624-0643-449d-90d4-95792935437e" path="/var/lib/kubelet/pods/1419a624-0643-449d-90d4-95792935437e/volumes"
Jul 24 17:08:52.031546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:52.031510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:08:52.092112 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:08:52.092089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:04.072023 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:04.071986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:09:04.091218 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:04.091196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:09:05.013143 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:05.013120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:09:16.793235 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:16.793149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:17.034627 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:17.034596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:19.419887 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:19.419850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:19.481541 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:19.481515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:27.327417 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:27.327386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:27.427963 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:27.427932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:27.989211 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:27.989177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:28.044891 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:28.044867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:30.402619 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:30.402583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:09:30.404996 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:30.404977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07768a9d-923d-4e34-9c7d-4db82a6cf6c2-metrics-certs\") pod \"network-metrics-daemon-w8hqg\" (UID: \"07768a9d-923d-4e34-9c7d-4db82a6cf6c2\") " pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:09:30.484728 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:30.484706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pgbn9\""
Jul 24 17:09:30.492630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:30.492612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8hqg"
Jul 24 17:09:30.622722 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:30.622686    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8hqg"]
Jul 24 17:09:30.624979 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:09:30.624946    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07768a9d_923d_4e34_9c7d_4db82a6cf6c2.slice/crio-f6b01c51062b3d07759677f6f0abefbb573ebe1721ebe241f8d5d3ce7179552f WatchSource:0}: Error finding container f6b01c51062b3d07759677f6f0abefbb573ebe1721ebe241f8d5d3ce7179552f: Status 404 returned error can't find the container with id f6b01c51062b3d07759677f6f0abefbb573ebe1721ebe241f8d5d3ce7179552f
Jul 24 17:09:30.627024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:30.627009    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:09:31.099051 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:31.099016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8hqg" event={"ID":"07768a9d-923d-4e34-9c7d-4db82a6cf6c2","Type":"ContainerStarted","Data":"f6b01c51062b3d07759677f6f0abefbb573ebe1721ebe241f8d5d3ce7179552f"}
Jul 24 17:09:32.026934 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:32.026909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:32.108866 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:32.108833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8hqg" event={"ID":"07768a9d-923d-4e34-9c7d-4db82a6cf6c2","Type":"ContainerStarted","Data":"08182e48ed763e856d3f2617e2a8b50f06e94c04a6ee39dd553126a1395f3bd0"}
Jul 24 17:09:32.109012 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:32.108921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8hqg" event={"ID":"07768a9d-923d-4e34-9c7d-4db82a6cf6c2","Type":"ContainerStarted","Data":"c483e756246b15a49e64083b8a489c9e1719a10821dd402a589355a16870e41f"}
Jul 24 17:09:32.124485 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:32.124439    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w8hqg" podStartSLOduration=374.270743107 podStartE2EDuration="6m15.124425773s" totalImagesPullingTime="853.682666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:09:32.123312804 +0000 UTC m=+375.954565499" watchObservedRunningTime="2026-07-24 17:09:32.124425773 +0000 UTC m=+375.955678460"
Jul 24 17:09:32.127272 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:32.127222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:32.228819 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:32.228794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:32.292752 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:32.292676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:33.235433 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:33.235399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:33.339668 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:33.339613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:34.524754 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:34.524720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:34.578703 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:34.578670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:35.122335 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:35.122310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:35.186323 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:35.186292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:44.019721 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:44.019691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:44.120614 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:44.120581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:45.590211 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:45.590176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:45.690888 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:45.690863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:45.769945 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:45.769927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:52.043740 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:52.043709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:52.145143 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:52.145112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:55.898065 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:55.898031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:55.998048 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:09:55.998020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:19.857249 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:19.857217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:19.959412 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:19.959384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:46.745614 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:46.745537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:47.582518 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:47.582490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:50.355823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:50.355793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:50.474320 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:50.474288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:51.171945 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:51.171916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:51.271084 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:51.271062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:52.235579 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:52.235554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:52.342335 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:52.342308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:54.242316 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:54.242281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:54.641586 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:54.641510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:57.150316 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:57.150284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:57.209957 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:10:57.209934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:02.160304 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:02.160270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:02.260967 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:02.260941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:03.045457 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:03.045431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:05.151471 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:05.151443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:05.254865 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:05.254832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:09.139898 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:09.139863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:09.238074 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:09.238047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:10.290801 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:10.290757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:11.366539 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:11.366510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:25.618181 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:25.618147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:25.725178 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:25.725153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:26.276541 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.276506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:26.553574 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.553498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:26.679935 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.679899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:26.778320 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.778290    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"]
Jul 24 17:11:26.778561 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.778550    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1419a624-0643-449d-90d4-95792935437e" containerName="console"
Jul 24 17:11:26.778610 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.778563    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1419a624-0643-449d-90d4-95792935437e" containerName="console"
Jul 24 17:11:26.778669 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.778611    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1419a624-0643-449d-90d4-95792935437e" containerName="console"
Jul 24 17:11:26.781464 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.781451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.784329 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.784304    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 17:11:26.784444 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.784416    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 17:11:26.784512 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.784488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 17:11:26.784599 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.784572    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 17:11:26.785047 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.785031    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pksz9\""
Jul 24 17:11:26.806337 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.806290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"]
Jul 24 17:11:26.830409 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.830390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-manifests\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.830496 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.830419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-tmp\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.830496 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.830449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.830496 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.830473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.830592 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.830518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8kk\" (UniqueName: \"kubernetes.io/projected/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-kube-api-access-hr8kk\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.931313 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.931284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-manifests\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.931418 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.931315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-tmp\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.931418 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.931363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.931418 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.931386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.931522 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.931429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8kk\" (UniqueName: \"kubernetes.io/projected/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-kube-api-access-hr8kk\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.931731 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.931713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-manifests\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.933822 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.933797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-tmp\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.933965 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.933950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.934026 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.933989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:26.948272 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:26.948251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8kk\" (UniqueName: \"kubernetes.io/projected/4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0-kube-api-access-hr8kk\") pod \"opendatahub-operator-controller-manager-645867896-kxdff\" (UID: \"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0\") " pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:27.047655 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:27.047631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:27.091287 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:27.091244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:27.234155 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:27.234127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:27.238783 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:27.238765    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"]
Jul 24 17:11:27.241279 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:11:27.241246    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e569e0f_fc4f_466f_81c9_fd9d4ff1d0f0.slice/crio-efa92a802e848ec3a4746e9c593816a5c9af3b6e706599fae2a3b941aabf50e5 WatchSource:0}: Error finding container efa92a802e848ec3a4746e9c593816a5c9af3b6e706599fae2a3b941aabf50e5: Status 404 returned error can't find the container with id efa92a802e848ec3a4746e9c593816a5c9af3b6e706599fae2a3b941aabf50e5
Jul 24 17:11:27.335934 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:27.335908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:27.432714 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:27.432639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:27.515816 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:27.515791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff" event={"ID":"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0","Type":"ContainerStarted","Data":"efa92a802e848ec3a4746e9c593816a5c9af3b6e706599fae2a3b941aabf50e5"}
Jul 24 17:11:30.533504 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:30.533471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff" event={"ID":"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0","Type":"ContainerStarted","Data":"1427aef58f201c913c515e9b8c028abe6eb12d36fa81bfabd302f762f3d7b9ed"}
Jul 24 17:11:31.538571 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:31.538535    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0" containerID="1427aef58f201c913c515e9b8c028abe6eb12d36fa81bfabd302f762f3d7b9ed" exitCode=0
Jul 24 17:11:31.538932 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:31.538614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff" event={"ID":"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0","Type":"ContainerDied","Data":"1427aef58f201c913c515e9b8c028abe6eb12d36fa81bfabd302f762f3d7b9ed"}
Jul 24 17:11:32.544985 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:32.544947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff" event={"ID":"4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0","Type":"ContainerStarted","Data":"b81fcfdf3aa9af72931a4dcbee14922eba9bc1ba5f5b1920bfdb6cdfe1ee12e3"}
Jul 24 17:11:32.545435 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:32.545121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:32.567818 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:32.567774    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff" podStartSLOduration=2.370988277 podStartE2EDuration="6.567723722s" totalImagesPullingTime="3.208538155s" totalInitContainerRuntime="988.19729ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:11:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:11:32.566855458 +0000 UTC m=+496.398108139" watchObservedRunningTime="2026-07-24 17:11:32.567723722 +0000 UTC m=+496.398976408"
Jul 24 17:11:41.478611 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:41.478576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:41.580271 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:41.580237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:43.550901 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:43.550874    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-645867896-kxdff"
Jul 24 17:11:45.542978 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.542944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:45.723313 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.723284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:45.740751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.740711    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"]
Jul 24 17:11:45.748860 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.748836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.753873 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.753808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 17:11:45.753873 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.753827    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 17:11:45.754568 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.754549    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 17:11:45.754840 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.754823    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:11:45.754840 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.754836    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pnpsn\""
Jul 24 17:11:45.755010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.754948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 17:11:45.759381 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.759359    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"]
Jul 24 17:11:45.872726 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.872633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-metrics-cert\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.872726 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.872707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrcz\" (UniqueName: \"kubernetes.io/projected/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-kube-api-access-rvrcz\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.872886 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.872732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-manager-config\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.872886 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.872754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-cert\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.973484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.973455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-manager-config\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.973484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.973487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-cert\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.973726 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.973542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-metrics-cert\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.973726 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.973571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrcz\" (UniqueName: \"kubernetes.io/projected/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-kube-api-access-rvrcz\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.974200 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.974176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-manager-config\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.975976 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.975945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-metrics-cert\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.976074 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.976010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-cert\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:45.990933 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:45.990911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrcz\" (UniqueName: \"kubernetes.io/projected/7eb9267f-5cf0-4536-bc3b-a40a775afdb5-kube-api-access-rvrcz\") pod \"lws-controller-manager-6f7bfdc884-bxnvh\" (UID: \"7eb9267f-5cf0-4536-bc3b-a40a775afdb5\") " pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:46.060051 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.060021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:46.220322 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.220301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"]
Jul 24 17:11:46.223104 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:11:46.223070    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb9267f_5cf0_4536_bc3b_a40a775afdb5.slice/crio-66377ad780a4f80874266ccd9b37088a4c3c013128da04c833191f5a670d944e WatchSource:0}: Error finding container 66377ad780a4f80874266ccd9b37088a4c3c013128da04c833191f5a670d944e: Status 404 returned error can't find the container with id 66377ad780a4f80874266ccd9b37088a4c3c013128da04c833191f5a670d944e
Jul 24 17:11:46.279940 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.279913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:46.384235 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.384208    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"]
Jul 24 17:11:46.388763 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.388746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.392204 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.392182    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 17:11:46.392297 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.392182    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 17:11:46.397818 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.397771    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 17:11:46.401945 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.401928    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fwj8w\""
Jul 24 17:11:46.402201 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.402190    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 17:11:46.415502 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.415478    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"]
Jul 24 17:11:46.476947 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.476921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f9199d-0dab-4be8-9021-3089cdd759da-tmp\") pod \"kube-auth-proxy-8cdf7544b-lwnng\" (UID: \"50f9199d-0dab-4be8-9021-3089cdd759da\") " pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.477117 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.476967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f9199d-0dab-4be8-9021-3089cdd759da-tls-certs\") pod \"kube-auth-proxy-8cdf7544b-lwnng\" (UID: \"50f9199d-0dab-4be8-9021-3089cdd759da\") " pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.477117 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.477037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrdm2\" (UniqueName: \"kubernetes.io/projected/50f9199d-0dab-4be8-9021-3089cdd759da-kube-api-access-jrdm2\") pod \"kube-auth-proxy-8cdf7544b-lwnng\" (UID: \"50f9199d-0dab-4be8-9021-3089cdd759da\") " pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.554301 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.554266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:46.578455 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.578430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f9199d-0dab-4be8-9021-3089cdd759da-tls-certs\") pod \"kube-auth-proxy-8cdf7544b-lwnng\" (UID: \"50f9199d-0dab-4be8-9021-3089cdd759da\") " pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.578612 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.578473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrdm2\" (UniqueName: \"kubernetes.io/projected/50f9199d-0dab-4be8-9021-3089cdd759da-kube-api-access-jrdm2\") pod \"kube-auth-proxy-8cdf7544b-lwnng\" (UID: \"50f9199d-0dab-4be8-9021-3089cdd759da\") " pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.578612 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.578506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f9199d-0dab-4be8-9021-3089cdd759da-tmp\") pod \"kube-auth-proxy-8cdf7544b-lwnng\" (UID: \"50f9199d-0dab-4be8-9021-3089cdd759da\") " pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.580944 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.580923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f9199d-0dab-4be8-9021-3089cdd759da-tmp\") pod \"kube-auth-proxy-8cdf7544b-lwnng\" (UID: \"50f9199d-0dab-4be8-9021-3089cdd759da\") " pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.581360 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.581340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f9199d-0dab-4be8-9021-3089cdd759da-tls-certs\") pod \"kube-auth-proxy-8cdf7544b-lwnng\" (UID: \"50f9199d-0dab-4be8-9021-3089cdd759da\") " pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.597402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.597374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh" event={"ID":"7eb9267f-5cf0-4536-bc3b-a40a775afdb5","Type":"ContainerStarted","Data":"66377ad780a4f80874266ccd9b37088a4c3c013128da04c833191f5a670d944e"}
Jul 24 17:11:46.597868 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.597848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrdm2\" (UniqueName: \"kubernetes.io/projected/50f9199d-0dab-4be8-9021-3089cdd759da-kube-api-access-jrdm2\") pod \"kube-auth-proxy-8cdf7544b-lwnng\" (UID: \"50f9199d-0dab-4be8-9021-3089cdd759da\") " pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.698538 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.698459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"
Jul 24 17:11:46.712923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.712881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:46.865538 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.865515    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng"]
Jul 24 17:11:46.867836 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:11:46.867809    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f9199d_0dab_4be8_9021_3089cdd759da.slice/crio-3a9abbc6fa5fc4e58bc01ccf41d1200d7d36b25c14b1333758957abfac0d6684 WatchSource:0}: Error finding container 3a9abbc6fa5fc4e58bc01ccf41d1200d7d36b25c14b1333758957abfac0d6684: Status 404 returned error can't find the container with id 3a9abbc6fa5fc4e58bc01ccf41d1200d7d36b25c14b1333758957abfac0d6684
Jul 24 17:11:46.955241 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:46.955157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:47.111295 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:47.111188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:47.501674 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:47.501620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:47.604892 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:47.604859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng" event={"ID":"50f9199d-0dab-4be8-9021-3089cdd759da","Type":"ContainerStarted","Data":"3a9abbc6fa5fc4e58bc01ccf41d1200d7d36b25c14b1333758957abfac0d6684"}
Jul 24 17:11:47.748668 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:47.748556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:48.269769 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:48.269739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:48.342409 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:48.342346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:49.360561 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:49.360524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:50.623894 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:50.623851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng" event={"ID":"50f9199d-0dab-4be8-9021-3089cdd759da","Type":"ContainerStarted","Data":"6eb557fe9cb73c458ee617a9314fa9290f41a33867b4968e16cd28ab5c38f1e8"}
Jul 24 17:11:50.626497 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:50.626463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh" event={"ID":"7eb9267f-5cf0-4536-bc3b-a40a775afdb5","Type":"ContainerStarted","Data":"10e18d94579aa4b41b57d3662ebba775e8913f0e4654fbcbb392f7078c8ef058"}
Jul 24 17:11:50.626990 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:50.626967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:11:50.644071 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:50.644015    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8cdf7544b-lwnng" podStartSLOduration=1.772386607 podStartE2EDuration="4.643997058s" totalImagesPullingTime="2.871610451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:11:50.641725022 +0000 UTC m=+514.472977710" watchObservedRunningTime="2026-07-24 17:11:50.643997058 +0000 UTC m=+514.475249750"
Jul 24 17:11:50.660207 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:50.660156    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh" podStartSLOduration=2.145500185 podStartE2EDuration="5.660135459s" totalImagesPullingTime="3.514635274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:11:50.657966443 +0000 UTC m=+514.489219143" watchObservedRunningTime="2026-07-24 17:11:50.660135459 +0000 UTC m=+514.491388147"
Jul 24 17:11:53.838069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:53.838039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:53.937085 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:53.937059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:54.241580 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:54.241528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:54.422789 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:54.422754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:54.515781 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:54.515722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:56.945877 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:56.945846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:57.073025 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:57.072991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:57.153014 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:57.152984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:58.241941 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:58.241911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:58.372630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:58.372599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:11:58.483144 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:11:58.483118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:12:02.638019 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:12:02.637987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f7bfdc884-bxnvh"
Jul 24 17:12:08.399754 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:12:08.399667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:12:08.499546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:12:08.499512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:12:47.294863 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:12:47.294824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:12:47.662817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:12:47.662737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:12:47.869881 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:12:47.869854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:12:48.184252 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:12:48.184224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:12:48.400144 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:12:48.400087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:12:54.073608 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:12:54.073557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:15.769125 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:15.769088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:16.577732 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:16.577700    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:13:16.578450 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:16.578419    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:13:19.622566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:19.622531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:22.569027 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:22.568990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:22.688450 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:22.688418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:24.147405 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:24.147377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:27.431547 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:27.431510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:27.586007 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:27.585977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:28.219483 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:28.219454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:28.373003 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:28.372973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:30.264239 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:30.264206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:32.043395 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:32.043367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:32.153484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:32.153454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:35.315822 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:35.315792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:35.536511 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:35.536488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:36.374920 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.374849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:36.385713 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.385686    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"]
Jul 24 17:13:36.389012 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.388998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:36.393186 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:36.393158    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"plugin-serving-cert\" is forbidden: User \"system:node:ip-10-0-140-80.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-80.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" type="*v1.Secret"
Jul 24 17:13:36.393735 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:36.393706    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kuadrant-console-nginx-conf\" is forbidden: User \"system:node:ip-10-0-140-80.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-80.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" type="*v1.ConfigMap"
Jul 24 17:13:36.393848 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:36.393821    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-80.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-80.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 24 17:13:36.393919 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:36.393854    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-console-plugin-66b5774985-kszgh\" is forbidden: User \"system:node:ip-10-0-140-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-80.ec2.internal' and this object" podUID="21904cd6-c74f-44e2-9ff5-c0f67a7e61cc" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:36.393981 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:36.393942    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-gttwz\" is forbidden: User \"system:node:ip-10-0-140-80.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-80.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gttwz\"" type="*v1.Secret"
Jul 24 17:13:36.394019 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:36.393976    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-80.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-80.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 24 17:13:36.429430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.429403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"]
Jul 24 17:13:36.498198 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.498175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqg2r\" (UniqueName: \"kubernetes.io/projected/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-kube-api-access-qqg2r\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:36.498290 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.498210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:36.498330 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.498310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:36.598126 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.598098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:36.599075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.599060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqg2r\" (UniqueName: \"kubernetes.io/projected/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-kube-api-access-qqg2r\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:36.599132 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.599088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:36.599132 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:36.599120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:37.205976 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:37.205900    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 17:13:37.206973 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:37.206956    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gttwz\""
Jul 24 17:13:37.525886 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:37.525861    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 17:13:37.528712 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:37.528686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqg2r\" (UniqueName: \"kubernetes.io/projected/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-kube-api-access-qqg2r\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:37.600215 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:37.600193    2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: failed to sync secret cache: timed out waiting for the condition
Jul 24 17:13:37.600215 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:37.600201    2563 configmap.go:193] Couldn't get configMap kuadrant-system/kuadrant-console-nginx-conf: failed to sync configmap cache: timed out waiting for the condition
Jul 24 17:13:37.600374 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:37.600284    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-nginx-conf podName:21904cd6-c74f-44e2-9ff5-c0f67a7e61cc nodeName:}" failed. No retries permitted until 2026-07-24 17:13:38.100263714 +0000 UTC m=+621.931516382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "nginx-conf" (UniqueName: "kubernetes.io/configmap/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-nginx-conf") pod "kuadrant-console-plugin-66b5774985-kszgh" (UID: "21904cd6-c74f-44e2-9ff5-c0f67a7e61cc") : failed to sync configmap cache: timed out waiting for the condition
Jul 24 17:13:37.600374 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:13:37.600307    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-plugin-serving-cert podName:21904cd6-c74f-44e2-9ff5-c0f67a7e61cc nodeName:}" failed. No retries permitted until 2026-07-24 17:13:38.10030025 +0000 UTC m=+621.931552920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-kszgh" (UID: "21904cd6-c74f-44e2-9ff5-c0f67a7e61cc") : failed to sync secret cache: timed out waiting for the condition
Jul 24 17:13:37.783574 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:37.783513    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 24 17:13:37.975179 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:37.975157    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 24 17:13:38.113238 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:38.113179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:38.113331 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:38.113244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:38.113873 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:38.113855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:38.115941 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:38.115922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21904cd6-c74f-44e2-9ff5-c0f67a7e61cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kszgh\" (UID: \"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:38.142900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:38.142868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:38.197711 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:38.197689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"
Jul 24 17:13:38.352722 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:38.352692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:38.367941 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:38.367882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh"]
Jul 24 17:13:38.370637 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:13:38.370614    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21904cd6_c74f_44e2_9ff5_c0f67a7e61cc.slice/crio-2aec3ce2e6d742cc3b4dcc624fa5d771ff2d365630b0a27ffbaa7865a869a829 WatchSource:0}: Error finding container 2aec3ce2e6d742cc3b4dcc624fa5d771ff2d365630b0a27ffbaa7865a869a829: Status 404 returned error can't find the container with id 2aec3ce2e6d742cc3b4dcc624fa5d771ff2d365630b0a27ffbaa7865a869a829
Jul 24 17:13:39.069785 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:39.069725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh" event={"ID":"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc","Type":"ContainerStarted","Data":"2aec3ce2e6d742cc3b4dcc624fa5d771ff2d365630b0a27ffbaa7865a869a829"}
Jul 24 17:13:39.132972 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:39.132940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:39.276111 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:39.276040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:40.746346 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:40.746315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:40.869108 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:40.869081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:48.117300 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:48.117264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:48.414729 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:48.414628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:48.556999 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:48.556963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:50.909259 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:50.909225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:51.213375 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:51.213301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:54.032221 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:54.032163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:54.214669 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:54.214617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:57.841401 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:57.841365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:58.005734 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:13:58.005698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:00.175428 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:00.175392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh" event={"ID":"21904cd6-c74f-44e2-9ff5-c0f67a7e61cc","Type":"ContainerStarted","Data":"553f3f6afa2411eedeca880301f9a624546ca9d9c5021fe7734ba808c8a90c50"}
Jul 24 17:14:00.205583 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:00.205536    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kszgh" podStartSLOduration=3.107179698 podStartE2EDuration="24.205521437s" totalImagesPullingTime="21.098341739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:00.205184 +0000 UTC m=+644.036436687" watchObservedRunningTime="2026-07-24 17:14:00.205521437 +0000 UTC m=+644.036774180"
Jul 24 17:14:08.439304 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:08.439268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:08.546305 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:08.546281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:08.643291 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:08.643256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:08.744872 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:08.744852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:19.452933 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:19.452901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:19.696706 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:19.696672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:21.997634 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:21.997602    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"]
Jul 24 17:14:21.998667 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:21.998628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:22.001043 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.001028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:22.003420 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.003403    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 17:14:22.009227 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.009209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"]
Jul 24 17:14:22.087779 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.087754    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"]
Jul 24 17:14:22.090999 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.090962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:22.160463 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.160438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23b4911-5485-4872-b0fa-fd580bd1e368-config-file\") pod \"limitador-limitador-64c94fd64f-mhpcl\" (UID: \"e23b4911-5485-4872-b0fa-fd580bd1e368\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:22.160547 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.160478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf8tq\" (UniqueName: \"kubernetes.io/projected/e23b4911-5485-4872-b0fa-fd580bd1e368-kube-api-access-jf8tq\") pod \"limitador-limitador-64c94fd64f-mhpcl\" (UID: \"e23b4911-5485-4872-b0fa-fd580bd1e368\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:22.167583 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.167559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:22.261846 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.261787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23b4911-5485-4872-b0fa-fd580bd1e368-config-file\") pod \"limitador-limitador-64c94fd64f-mhpcl\" (UID: \"e23b4911-5485-4872-b0fa-fd580bd1e368\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:22.261846 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.261827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf8tq\" (UniqueName: \"kubernetes.io/projected/e23b4911-5485-4872-b0fa-fd580bd1e368-kube-api-access-jf8tq\") pod \"limitador-limitador-64c94fd64f-mhpcl\" (UID: \"e23b4911-5485-4872-b0fa-fd580bd1e368\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:22.262471 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.262451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23b4911-5485-4872-b0fa-fd580bd1e368-config-file\") pod \"limitador-limitador-64c94fd64f-mhpcl\" (UID: \"e23b4911-5485-4872-b0fa-fd580bd1e368\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:22.273046 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.273018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf8tq\" (UniqueName: \"kubernetes.io/projected/e23b4911-5485-4872-b0fa-fd580bd1e368-kube-api-access-jf8tq\") pod \"limitador-limitador-64c94fd64f-mhpcl\" (UID: \"e23b4911-5485-4872-b0fa-fd580bd1e368\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:22.311942 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.311916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:22.438957 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.438850    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"]
Jul 24 17:14:22.441675 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:14:22.441626    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23b4911_5485_4872_b0fa_fd580bd1e368.slice/crio-8b7779c3d7ebe04ea09e5a3ad9cf994251c73cd63bef520aec94b17ff66b7483 WatchSource:0}: Error finding container 8b7779c3d7ebe04ea09e5a3ad9cf994251c73cd63bef520aec94b17ff66b7483: Status 404 returned error can't find the container with id 8b7779c3d7ebe04ea09e5a3ad9cf994251c73cd63bef520aec94b17ff66b7483
Jul 24 17:14:22.815867 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.815841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:22.905989 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:22.905959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:23.275525 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:23.275483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl" event={"ID":"e23b4911-5485-4872-b0fa-fd580bd1e368","Type":"ContainerStarted","Data":"8b7779c3d7ebe04ea09e5a3ad9cf994251c73cd63bef520aec94b17ff66b7483"}
Jul 24 17:14:25.286981 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:25.286899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl" event={"ID":"e23b4911-5485-4872-b0fa-fd580bd1e368","Type":"ContainerStarted","Data":"e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa"}
Jul 24 17:14:25.287324 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:25.287032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:25.310137 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:25.310077    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl" podStartSLOduration=1.749181338 podStartE2EDuration="4.310054216s" totalImagesPullingTime="2.560872878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:25.308982544 +0000 UTC m=+669.140235232" watchObservedRunningTime="2026-07-24 17:14:25.310054216 +0000 UTC m=+669.141306904"
Jul 24 17:14:36.292616 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:36.292585    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:36.536136 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:36.536105    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"]
Jul 24 17:14:36.536285 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:36.536265    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl" podUID="e23b4911-5485-4872-b0fa-fd580bd1e368" containerName="limitador" containerID="cri-o://e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa" gracePeriod=30
Jul 24 17:14:37.075452 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.075433    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:37.171323 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.171259    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23b4911-5485-4872-b0fa-fd580bd1e368-config-file\") pod \"e23b4911-5485-4872-b0fa-fd580bd1e368\" (UID: \"e23b4911-5485-4872-b0fa-fd580bd1e368\") "
Jul 24 17:14:37.171323 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.171309    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf8tq\" (UniqueName: \"kubernetes.io/projected/e23b4911-5485-4872-b0fa-fd580bd1e368-kube-api-access-jf8tq\") pod \"e23b4911-5485-4872-b0fa-fd580bd1e368\" (UID: \"e23b4911-5485-4872-b0fa-fd580bd1e368\") "
Jul 24 17:14:37.171673 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.171630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23b4911-5485-4872-b0fa-fd580bd1e368-config-file" (OuterVolumeSpecName: "config-file") pod "e23b4911-5485-4872-b0fa-fd580bd1e368" (UID: "e23b4911-5485-4872-b0fa-fd580bd1e368"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:14:37.173396 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.173373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23b4911-5485-4872-b0fa-fd580bd1e368-kube-api-access-jf8tq" (OuterVolumeSpecName: "kube-api-access-jf8tq") pod "e23b4911-5485-4872-b0fa-fd580bd1e368" (UID: "e23b4911-5485-4872-b0fa-fd580bd1e368"). InnerVolumeSpecName "kube-api-access-jf8tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:14:37.272697 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.272677    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23b4911-5485-4872-b0fa-fd580bd1e368-config-file\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:14:37.272697 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.272696    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf8tq\" (UniqueName: \"kubernetes.io/projected/e23b4911-5485-4872-b0fa-fd580bd1e368-kube-api-access-jf8tq\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:14:37.337420 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.337387    2563 generic.go:356] "Generic (PLEG): container finished" podID="e23b4911-5485-4872-b0fa-fd580bd1e368" containerID="e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa" exitCode=0
Jul 24 17:14:37.337722 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.337428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl" event={"ID":"e23b4911-5485-4872-b0fa-fd580bd1e368","Type":"ContainerDied","Data":"e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa"}
Jul 24 17:14:37.337722 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.337431    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"
Jul 24 17:14:37.337722 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.337469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mhpcl" event={"ID":"e23b4911-5485-4872-b0fa-fd580bd1e368","Type":"ContainerDied","Data":"8b7779c3d7ebe04ea09e5a3ad9cf994251c73cd63bef520aec94b17ff66b7483"}
Jul 24 17:14:37.337722 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.337493    2563 scope.go:117] "RemoveContainer" containerID="e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa"
Jul 24 17:14:37.345656 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.345629    2563 scope.go:117] "RemoveContainer" containerID="e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa"
Jul 24 17:14:37.345944 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:14:37.345925    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa\": container with ID starting with e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa not found: ID does not exist" containerID="e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa"
Jul 24 17:14:37.346004 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.345950    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa"} err="failed to get container status \"e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa\": rpc error: code = NotFound desc = could not find container \"e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa\": container with ID starting with e45f379a0e259eeb251ba0fac9228f63a4973be6c9d83501c7ac4aed4ae5d5aa not found: ID does not exist"
Jul 24 17:14:37.373440 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.373422    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"]
Jul 24 17:14:37.380278 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:37.380259    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mhpcl"]
Jul 24 17:14:38.331300 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.331269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:38.489152 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.489128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:38.687386 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.687314    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23b4911-5485-4872-b0fa-fd580bd1e368" path="/var/lib/kubelet/pods/e23b4911-5485-4872-b0fa-fd580bd1e368/volumes"
Jul 24 17:14:38.804669 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.804624    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"]
Jul 24 17:14:38.804957 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.804945    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23b4911-5485-4872-b0fa-fd580bd1e368" containerName="limitador"
Jul 24 17:14:38.805015 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.804958    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23b4911-5485-4872-b0fa-fd580bd1e368" containerName="limitador"
Jul 24 17:14:38.805015 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.805006    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23b4911-5485-4872-b0fa-fd580bd1e368" containerName="limitador"
Jul 24 17:14:38.809271 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.809253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:38.814125 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.814107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 17:14:38.814354 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.814332    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 17:14:38.814868 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.814853    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mxjq2\""
Jul 24 17:14:38.815160 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.815146    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 17:14:38.847976 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.847956    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"]
Jul 24 17:14:38.924812 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.924795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:38.982804 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.982780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8vvb\" (UniqueName: \"kubernetes.io/projected/2d74ad54-5972-4531-860a-857d62cff6b2-kube-api-access-w8vvb\") pod \"postgres-69fb4594cf-xmjxf\" (UID: \"2d74ad54-5972-4531-860a-857d62cff6b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:38.983010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:38.982825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2d74ad54-5972-4531-860a-857d62cff6b2-data\") pod \"postgres-69fb4594cf-xmjxf\" (UID: \"2d74ad54-5972-4531-860a-857d62cff6b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:39.084824 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:39.084787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8vvb\" (UniqueName: \"kubernetes.io/projected/2d74ad54-5972-4531-860a-857d62cff6b2-kube-api-access-w8vvb\") pod \"postgres-69fb4594cf-xmjxf\" (UID: \"2d74ad54-5972-4531-860a-857d62cff6b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:39.084974 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:39.084839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2d74ad54-5972-4531-860a-857d62cff6b2-data\") pod \"postgres-69fb4594cf-xmjxf\" (UID: \"2d74ad54-5972-4531-860a-857d62cff6b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:39.085188 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:39.085172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2d74ad54-5972-4531-860a-857d62cff6b2-data\") pod \"postgres-69fb4594cf-xmjxf\" (UID: \"2d74ad54-5972-4531-860a-857d62cff6b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:39.095813 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:39.095790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8vvb\" (UniqueName: \"kubernetes.io/projected/2d74ad54-5972-4531-860a-857d62cff6b2-kube-api-access-w8vvb\") pod \"postgres-69fb4594cf-xmjxf\" (UID: \"2d74ad54-5972-4531-860a-857d62cff6b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:39.118886 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:39.118858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:39.271762 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:39.271686    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"]
Jul 24 17:14:39.274669 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:14:39.274619    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d74ad54_5972_4531_860a_857d62cff6b2.slice/crio-6aaec2789e4af7c2768a8e5b61521bd6caa79a9fbe3f46053b174d5e2d319a93 WatchSource:0}: Error finding container 6aaec2789e4af7c2768a8e5b61521bd6caa79a9fbe3f46053b174d5e2d319a93: Status 404 returned error can't find the container with id 6aaec2789e4af7c2768a8e5b61521bd6caa79a9fbe3f46053b174d5e2d319a93
Jul 24 17:14:39.275918 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:39.275898    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:14:39.348449 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:39.348416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf" event={"ID":"2d74ad54-5972-4531-860a-857d62cff6b2","Type":"ContainerStarted","Data":"6aaec2789e4af7c2768a8e5b61521bd6caa79a9fbe3f46053b174d5e2d319a93"}
Jul 24 17:14:44.374431 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:44.374403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf" event={"ID":"2d74ad54-5972-4531-860a-857d62cff6b2","Type":"ContainerStarted","Data":"4028c0d75275279d7fe3450ecf8e35246bb5dd3d3e2cc30f0c14cfe2c706a1e5"}
Jul 24 17:14:44.374854 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:44.374472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:44.391786 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:44.391742    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf" podStartSLOduration=1.478942273 podStartE2EDuration="6.391728905s" totalImagesPullingTime="4.912786632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:44.390372894 +0000 UTC m=+688.221625580" watchObservedRunningTime="2026-07-24 17:14:44.391728905 +0000 UTC m=+688.222981591"
Jul 24 17:14:50.408290 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:50.408264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xmjxf"
Jul 24 17:14:50.726745 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:50.726718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:50.994065 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:50.994007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:54.402707 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:54.402675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:58.113258 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.113228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:58.136898 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.136872    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86d7b47df5-c5x9q"]
Jul 24 17:14:58.215047 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.215027    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86d7b47df5-c5x9q"]
Jul 24 17:14:58.215173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.215109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:14:58.219215 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.219193    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:14:58.219336 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.219198    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5lmsg\""
Jul 24 17:14:58.219336 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.219198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:14:58.223742 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.223725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64d4\" (UniqueName: \"kubernetes.io/projected/43817637-ad4d-43d0-b029-b094453c1752-kube-api-access-x64d4\") pod \"maas-controller-86d7b47df5-c5x9q\" (UID: \"43817637-ad4d-43d0-b029-b094453c1752\") " pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:14:58.223850 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.223774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43817637-ad4d-43d0-b029-b094453c1752-webhook-cert\") pod \"maas-controller-86d7b47df5-c5x9q\" (UID: \"43817637-ad4d-43d0-b029-b094453c1752\") " pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:14:58.282600 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.282582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:58.324928 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.324908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43817637-ad4d-43d0-b029-b094453c1752-webhook-cert\") pod \"maas-controller-86d7b47df5-c5x9q\" (UID: \"43817637-ad4d-43d0-b029-b094453c1752\") " pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:14:58.325012 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.324949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64d4\" (UniqueName: \"kubernetes.io/projected/43817637-ad4d-43d0-b029-b094453c1752-kube-api-access-x64d4\") pod \"maas-controller-86d7b47df5-c5x9q\" (UID: \"43817637-ad4d-43d0-b029-b094453c1752\") " pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:14:58.325059 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:14:58.325037    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 24 17:14:58.325100 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:14:58.325093    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43817637-ad4d-43d0-b029-b094453c1752-webhook-cert podName:43817637-ad4d-43d0-b029-b094453c1752 nodeName:}" failed. No retries permitted until 2026-07-24 17:14:58.825077612 +0000 UTC m=+702.656330276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/43817637-ad4d-43d0-b029-b094453c1752-webhook-cert") pod "maas-controller-86d7b47df5-c5x9q" (UID: "43817637-ad4d-43d0-b029-b094453c1752") : secret "maas-controller-webhook-cert" not found
Jul 24 17:14:58.334857 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.334836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64d4\" (UniqueName: \"kubernetes.io/projected/43817637-ad4d-43d0-b029-b094453c1752-kube-api-access-x64d4\") pod \"maas-controller-86d7b47df5-c5x9q\" (UID: \"43817637-ad4d-43d0-b029-b094453c1752\") " pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:14:58.380667 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.380595    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-87594f4b5-nfcg4"]
Jul 24 17:14:58.383582 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.383564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:58.406306 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.406268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-87594f4b5-nfcg4"]
Jul 24 17:14:58.406402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.406363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:14:58.426073 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.426053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5978becc-dbce-465f-8efc-d3f39dbb3ccc-webhook-cert\") pod \"maas-controller-87594f4b5-nfcg4\" (UID: \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\") " pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:14:58.426153 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.426096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbrr\" (UniqueName: \"kubernetes.io/projected/5978becc-dbce-465f-8efc-d3f39dbb3ccc-kube-api-access-hwbrr\") pod \"maas-controller-87594f4b5-nfcg4\" (UID: \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\") " pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:14:58.492503 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.492483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:58.526655 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.526624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5978becc-dbce-465f-8efc-d3f39dbb3ccc-webhook-cert\") pod \"maas-controller-87594f4b5-nfcg4\" (UID: \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\") " pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:14:58.526742 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.526692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbrr\" (UniqueName: \"kubernetes.io/projected/5978becc-dbce-465f-8efc-d3f39dbb3ccc-kube-api-access-hwbrr\") pod \"maas-controller-87594f4b5-nfcg4\" (UID: \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\") " pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:14:58.529034 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.529016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5978becc-dbce-465f-8efc-d3f39dbb3ccc-webhook-cert\") pod \"maas-controller-87594f4b5-nfcg4\" (UID: \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\") " pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:14:58.535362 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.535341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbrr\" (UniqueName: \"kubernetes.io/projected/5978becc-dbce-465f-8efc-d3f39dbb3ccc-kube-api-access-hwbrr\") pod \"maas-controller-87594f4b5-nfcg4\" (UID: \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\") " pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:14:58.716629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.716608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:14:58.829510 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.829475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43817637-ad4d-43d0-b029-b094453c1752-webhook-cert\") pod \"maas-controller-86d7b47df5-c5x9q\" (UID: \"43817637-ad4d-43d0-b029-b094453c1752\") " pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:14:58.832059 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.832036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43817637-ad4d-43d0-b029-b094453c1752-webhook-cert\") pod \"maas-controller-86d7b47df5-c5x9q\" (UID: \"43817637-ad4d-43d0-b029-b094453c1752\") " pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:14:58.875499 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:58.875475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-87594f4b5-nfcg4"]
Jul 24 17:14:58.878004 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:14:58.877972    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5978becc_dbce_465f_8efc_d3f39dbb3ccc.slice/crio-ac1eef67c03082c40bfa16e0963b2909e13fec97c70cef76cfda4f153df97674 WatchSource:0}: Error finding container ac1eef67c03082c40bfa16e0963b2909e13fec97c70cef76cfda4f153df97674: Status 404 returned error can't find the container with id ac1eef67c03082c40bfa16e0963b2909e13fec97c70cef76cfda4f153df97674
Jul 24 17:14:59.124798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:59.124734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:14:59.312116 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:59.312084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86d7b47df5-c5x9q"]
Jul 24 17:14:59.314020 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:14:59.313996    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43817637_ad4d_43d0_b029_b094453c1752.slice/crio-fe13db45b271e395f526bd3c2ce1c722ff37384633b1014707e9fc90961169f7 WatchSource:0}: Error finding container fe13db45b271e395f526bd3c2ce1c722ff37384633b1014707e9fc90961169f7: Status 404 returned error can't find the container with id fe13db45b271e395f526bd3c2ce1c722ff37384633b1014707e9fc90961169f7
Jul 24 17:14:59.439434 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:59.439358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-nfcg4" event={"ID":"5978becc-dbce-465f-8efc-d3f39dbb3ccc","Type":"ContainerStarted","Data":"ac1eef67c03082c40bfa16e0963b2909e13fec97c70cef76cfda4f153df97674"}
Jul 24 17:14:59.440401 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:14:59.440380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d7b47df5-c5x9q" event={"ID":"43817637-ad4d-43d0-b029-b094453c1752","Type":"ContainerStarted","Data":"fe13db45b271e395f526bd3c2ce1c722ff37384633b1014707e9fc90961169f7"}
Jul 24 17:15:01.670406 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:01.670375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:01.782489 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:01.782458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:02.456711 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:02.456665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d7b47df5-c5x9q" event={"ID":"43817637-ad4d-43d0-b029-b094453c1752","Type":"ContainerStarted","Data":"d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d"}
Jul 24 17:15:02.456882 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:02.456729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:15:02.458032 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:02.458010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-nfcg4" event={"ID":"5978becc-dbce-465f-8efc-d3f39dbb3ccc","Type":"ContainerStarted","Data":"0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2"}
Jul 24 17:15:02.458154 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:02.458141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:15:02.488056 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:02.488001    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86d7b47df5-c5x9q" podStartSLOduration=2.3629857149999998 podStartE2EDuration="4.48798794s" totalImagesPullingTime="2.125002225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:15:02.485865607 +0000 UTC m=+706.317118294" watchObservedRunningTime="2026-07-24 17:15:02.48798794 +0000 UTC m=+706.319240674"
Jul 24 17:15:02.516697 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:02.516631    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-87594f4b5-nfcg4" podStartSLOduration=1.9621665 podStartE2EDuration="4.516616798s" totalImagesPullingTime="2.554450298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:15:02.513553553 +0000 UTC m=+706.344806239" watchObservedRunningTime="2026-07-24 17:15:02.516616798 +0000 UTC m=+706.347869483"
Jul 24 17:15:13.463967 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.463891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:15:13.463967 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.463955    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:15:13.523958 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.523930    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86d7b47df5-c5x9q"]
Jul 24 17:15:13.524136 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.524114    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86d7b47df5-c5x9q" podUID="43817637-ad4d-43d0-b029-b094453c1752" containerName="manager" containerID="cri-o://d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d" gracePeriod=30
Jul 24 17:15:13.770610 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.770584    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:15:13.840350 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.840328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43817637-ad4d-43d0-b029-b094453c1752-webhook-cert\") pod \"43817637-ad4d-43d0-b029-b094453c1752\" (UID: \"43817637-ad4d-43d0-b029-b094453c1752\") "
Jul 24 17:15:13.840448 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.840396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x64d4\" (UniqueName: \"kubernetes.io/projected/43817637-ad4d-43d0-b029-b094453c1752-kube-api-access-x64d4\") pod \"43817637-ad4d-43d0-b029-b094453c1752\" (UID: \"43817637-ad4d-43d0-b029-b094453c1752\") "
Jul 24 17:15:13.842509 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.842480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43817637-ad4d-43d0-b029-b094453c1752-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "43817637-ad4d-43d0-b029-b094453c1752" (UID: "43817637-ad4d-43d0-b029-b094453c1752"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:15:13.842598 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.842524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43817637-ad4d-43d0-b029-b094453c1752-kube-api-access-x64d4" (OuterVolumeSpecName: "kube-api-access-x64d4") pod "43817637-ad4d-43d0-b029-b094453c1752" (UID: "43817637-ad4d-43d0-b029-b094453c1752"). InnerVolumeSpecName "kube-api-access-x64d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:15:13.940795 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.940773    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43817637-ad4d-43d0-b029-b094453c1752-webhook-cert\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:15:13.940795 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:13.940792    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x64d4\" (UniqueName: \"kubernetes.io/projected/43817637-ad4d-43d0-b029-b094453c1752-kube-api-access-x64d4\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:15:14.508009 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.507978    2563 generic.go:356] "Generic (PLEG): container finished" podID="43817637-ad4d-43d0-b029-b094453c1752" containerID="d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d" exitCode=0
Jul 24 17:15:14.508379 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.508021    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d7b47df5-c5x9q"
Jul 24 17:15:14.508379 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.508069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d7b47df5-c5x9q" event={"ID":"43817637-ad4d-43d0-b029-b094453c1752","Type":"ContainerDied","Data":"d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d"}
Jul 24 17:15:14.508379 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.508146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d7b47df5-c5x9q" event={"ID":"43817637-ad4d-43d0-b029-b094453c1752","Type":"ContainerDied","Data":"fe13db45b271e395f526bd3c2ce1c722ff37384633b1014707e9fc90961169f7"}
Jul 24 17:15:14.508379 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.508175    2563 scope.go:117] "RemoveContainer" containerID="d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d"
Jul 24 17:15:14.519953 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.519938    2563 scope.go:117] "RemoveContainer" containerID="d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d"
Jul 24 17:15:14.520211 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:15:14.520194    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d\": container with ID starting with d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d not found: ID does not exist" containerID="d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d"
Jul 24 17:15:14.520267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.520221    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d"} err="failed to get container status \"d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d\": rpc error: code = NotFound desc = could not find container \"d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d\": container with ID starting with d009e284399e44420ac236719ac287e0ffada8c3ce9cf1f84c83be141b5a426d not found: ID does not exist"
Jul 24 17:15:14.530902 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.530879    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86d7b47df5-c5x9q"]
Jul 24 17:15:14.533821 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.533802    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86d7b47df5-c5x9q"]
Jul 24 17:15:14.688126 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:14.688103    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43817637-ad4d-43d0-b029-b094453c1752" path="/var/lib/kubelet/pods/43817637-ad4d-43d0-b029-b094453c1752/volumes"
Jul 24 17:15:32.558689 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:32.558616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:32.655119 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:32.655094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:33.195792 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.195763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:33.730846 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.730821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:33.853996 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.853971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:33.860366 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.860343    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"]
Jul 24 17:15:33.863383 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.861572    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43817637-ad4d-43d0-b029-b094453c1752" containerName="manager"
Jul 24 17:15:33.863383 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.861623    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43817637-ad4d-43d0-b029-b094453c1752" containerName="manager"
Jul 24 17:15:33.863383 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.861791    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="43817637-ad4d-43d0-b029-b094453c1752" containerName="manager"
Jul 24 17:15:33.871216 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.871195    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:33.873893 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.873872    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 17:15:33.880202 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.880181    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8n49r"]
Jul 24 17:15:33.884097 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.884076    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"]
Jul 24 17:15:33.884204 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.884186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:33.917280 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.917258    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8n49r"]
Jul 24 17:15:33.978714 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.978692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4vg\" (UniqueName: \"kubernetes.io/projected/2ba6c789-6276-471c-8198-999aa9dfab2a-kube-api-access-8s4vg\") pod \"payload-pre-processing-687dbb6cf6-zbfnf\" (UID: \"2ba6c789-6276-471c-8198-999aa9dfab2a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:33.978817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.978719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af7f2dd-29f5-43b5-964f-95142729e455-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8n49r\" (UID: \"9af7f2dd-29f5-43b5-964f-95142729e455\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:33.978817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.978736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5jwf\" (UniqueName: \"kubernetes.io/projected/9af7f2dd-29f5-43b5-964f-95142729e455-kube-api-access-w5jwf\") pod \"payload-processing-5cc6cd444b-8n49r\" (UID: \"9af7f2dd-29f5-43b5-964f-95142729e455\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:33.978817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:33.978758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba6c789-6276-471c-8198-999aa9dfab2a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zbfnf\" (UID: \"2ba6c789-6276-471c-8198-999aa9dfab2a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:34.055890 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.055823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:34.080136 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.080115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4vg\" (UniqueName: \"kubernetes.io/projected/2ba6c789-6276-471c-8198-999aa9dfab2a-kube-api-access-8s4vg\") pod \"payload-pre-processing-687dbb6cf6-zbfnf\" (UID: \"2ba6c789-6276-471c-8198-999aa9dfab2a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:34.080249 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.080234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af7f2dd-29f5-43b5-964f-95142729e455-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8n49r\" (UID: \"9af7f2dd-29f5-43b5-964f-95142729e455\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:34.080313 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.080262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jwf\" (UniqueName: \"kubernetes.io/projected/9af7f2dd-29f5-43b5-964f-95142729e455-kube-api-access-w5jwf\") pod \"payload-processing-5cc6cd444b-8n49r\" (UID: \"9af7f2dd-29f5-43b5-964f-95142729e455\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:34.080313 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.080293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba6c789-6276-471c-8198-999aa9dfab2a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zbfnf\" (UID: \"2ba6c789-6276-471c-8198-999aa9dfab2a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:34.080931 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.080912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af7f2dd-29f5-43b5-964f-95142729e455-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8n49r\" (UID: \"9af7f2dd-29f5-43b5-964f-95142729e455\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:34.081060 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.081043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba6c789-6276-471c-8198-999aa9dfab2a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zbfnf\" (UID: \"2ba6c789-6276-471c-8198-999aa9dfab2a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:34.088971 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.088939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4vg\" (UniqueName: \"kubernetes.io/projected/2ba6c789-6276-471c-8198-999aa9dfab2a-kube-api-access-8s4vg\") pod \"payload-pre-processing-687dbb6cf6-zbfnf\" (UID: \"2ba6c789-6276-471c-8198-999aa9dfab2a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:34.089252 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.089237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jwf\" (UniqueName: \"kubernetes.io/projected/9af7f2dd-29f5-43b5-964f-95142729e455-kube-api-access-w5jwf\") pod \"payload-processing-5cc6cd444b-8n49r\" (UID: \"9af7f2dd-29f5-43b5-964f-95142729e455\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:34.182348 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.182325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:34.192985 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.192918    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:34.328981 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.328901    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"]
Jul 24 17:15:34.332557 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:15:34.332516    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba6c789_6276_471c_8198_999aa9dfab2a.slice/crio-7767674c02f2449c221283900f3463146e3e7d5a586ec5e570076c4652e97fbc WatchSource:0}: Error finding container 7767674c02f2449c221283900f3463146e3e7d5a586ec5e570076c4652e97fbc: Status 404 returned error can't find the container with id 7767674c02f2449c221283900f3463146e3e7d5a586ec5e570076c4652e97fbc
Jul 24 17:15:34.346161 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.346129    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8n49r"]
Jul 24 17:15:34.348984 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:15:34.348959    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af7f2dd_29f5_43b5_964f_95142729e455.slice/crio-2a30863b1659698c645ec323f4f453a426fa308e533445dd46e8fba49daf79d0 WatchSource:0}: Error finding container 2a30863b1659698c645ec323f4f453a426fa308e533445dd46e8fba49daf79d0: Status 404 returned error can't find the container with id 2a30863b1659698c645ec323f4f453a426fa308e533445dd46e8fba49daf79d0
Jul 24 17:15:34.601135 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.601077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf" event={"ID":"2ba6c789-6276-471c-8198-999aa9dfab2a","Type":"ContainerStarted","Data":"7767674c02f2449c221283900f3463146e3e7d5a586ec5e570076c4652e97fbc"}
Jul 24 17:15:34.602202 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:34.602181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r" event={"ID":"9af7f2dd-29f5-43b5-964f-95142729e455","Type":"ContainerStarted","Data":"2a30863b1659698c645ec323f4f453a426fa308e533445dd46e8fba49daf79d0"}
Jul 24 17:15:37.620193 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:37.620160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf" event={"ID":"2ba6c789-6276-471c-8198-999aa9dfab2a","Type":"ContainerStarted","Data":"d3cc47f7821a4b017a6832a491c4f5439ea128895d3ed3f983438bc96f7cfd58"}
Jul 24 17:15:37.620624 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:37.620226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:37.621719 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:37.621697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r" event={"ID":"9af7f2dd-29f5-43b5-964f-95142729e455","Type":"ContainerStarted","Data":"f7404f03ccb8fd04655955034b439c421baad4aeebeb0aba45348fc9a5134b94"}
Jul 24 17:15:37.621841 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:37.621826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:37.635251 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:37.635204    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf" podStartSLOduration=2.214400286 podStartE2EDuration="4.635187233s" totalImagesPullingTime="2.420786947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:15:37.634318591 +0000 UTC m=+741.465571277" watchObservedRunningTime="2026-07-24 17:15:37.635187233 +0000 UTC m=+741.466439922"
Jul 24 17:15:37.654940 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:37.654902    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r" podStartSLOduration=2.243734433 podStartE2EDuration="4.6548908s" totalImagesPullingTime="2.411156367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:15:37.653200292 +0000 UTC m=+741.484452980" watchObservedRunningTime="2026-07-24 17:15:37.6548908 +0000 UTC m=+741.486143486"
Jul 24 17:15:46.456580 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:46.456548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:46.554069 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:46.554041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:48.627704 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:48.627674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zbfnf"
Jul 24 17:15:48.628145 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:48.627932    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8n49r"
Jul 24 17:15:54.712666 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:15:54.712608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:05.088361 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:05.088330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:28.447495 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.447463    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"]
Jul 24 17:16:28.451354 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.451334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.453685 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.453663    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 17:16:28.453813 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.453714    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 17:16:28.453813 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.453733    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 24 17:16:28.454338 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.454319    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6kmdn\""
Jul 24 17:16:28.460927 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.460907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"]
Jul 24 17:16:28.481188 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.481156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:28.490086 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.490064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.490194 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.490103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpnt\" (UniqueName: \"kubernetes.io/projected/b490414c-4d11-4035-a522-7c5c869098b0-kube-api-access-dxpnt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.490194 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.490127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.490277 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.490194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.490277 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.490234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b490414c-4d11-4035-a522-7c5c869098b0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.490277 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.490269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.591147 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.591112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.591348 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.591172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.591348 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.591205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpnt\" (UniqueName: \"kubernetes.io/projected/b490414c-4d11-4035-a522-7c5c869098b0-kube-api-access-dxpnt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.591348 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.591235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.591348 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.591281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.591348 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.591326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b490414c-4d11-4035-a522-7c5c869098b0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.591617 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.591495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.591716 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.591613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.591809 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.591793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.593716 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.593685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b490414c-4d11-4035-a522-7c5c869098b0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.594000 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.593975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b490414c-4d11-4035-a522-7c5c869098b0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.599352 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.599332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpnt\" (UniqueName: \"kubernetes.io/projected/b490414c-4d11-4035-a522-7c5c869098b0-kube-api-access-dxpnt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qtlkw\" (UID: \"b490414c-4d11-4035-a522-7c5c869098b0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.762221 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.762182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:28.892497 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:28.892342    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"]
Jul 24 17:16:28.895268 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:16:28.895244    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb490414c_4d11_4035_a522_7c5c869098b0.slice/crio-b0496044f8a3d11e11bfc38f80d052efb00f5c37fec9ed130944442af64f6c5a WatchSource:0}: Error finding container b0496044f8a3d11e11bfc38f80d052efb00f5c37fec9ed130944442af64f6c5a: Status 404 returned error can't find the container with id b0496044f8a3d11e11bfc38f80d052efb00f5c37fec9ed130944442af64f6c5a
Jul 24 17:16:29.848546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:29.848506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw" event={"ID":"b490414c-4d11-4035-a522-7c5c869098b0","Type":"ContainerStarted","Data":"b0496044f8a3d11e11bfc38f80d052efb00f5c37fec9ed130944442af64f6c5a"}
Jul 24 17:16:30.853969 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:30.853939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw" event={"ID":"b490414c-4d11-4035-a522-7c5c869098b0","Type":"ContainerStarted","Data":"46e31f719a6e6c3da21b85b6e5c2ea48e125a80b3e4f8b399f0c9db012366870"}
Jul 24 17:16:30.870390 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:30.870349    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw" podStartSLOduration=1.8025371159999999 podStartE2EDuration="2.870312151s" totalImagesPullingTime="1.067775035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:16:30.869525513 +0000 UTC m=+794.700778200" watchObservedRunningTime="2026-07-24 17:16:30.870312151 +0000 UTC m=+794.701564842"
Jul 24 17:16:38.762416 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:38.762345    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:38.762416 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:38.762403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:38.775046 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:38.775024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:38.898922 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:38.898896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qtlkw"
Jul 24 17:16:43.445058 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.445027    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"]
Jul 24 17:16:43.447958 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.447941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.450285 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.450266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 24 17:16:43.459009 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.458987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"]
Jul 24 17:16:43.473312 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.473290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:43.510095 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.510069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.510188 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.510109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a11148e-837d-4b22-9b37-9b55b52b4714-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.510259 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.510191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.510259 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.510228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.510363 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.510287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzqfg\" (UniqueName: \"kubernetes.io/projected/3a11148e-837d-4b22-9b37-9b55b52b4714-kube-api-access-fzqfg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.510363 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.510334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.610901 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.610865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.610901 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.610909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a11148e-837d-4b22-9b37-9b55b52b4714-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.611084 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.610955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.611084 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.610974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.611084 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.611003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzqfg\" (UniqueName: \"kubernetes.io/projected/3a11148e-837d-4b22-9b37-9b55b52b4714-kube-api-access-fzqfg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.611084 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.611032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.611338 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.611316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.611401 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.611337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.611401 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.611381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.613456 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.613431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a11148e-837d-4b22-9b37-9b55b52b4714-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.613583 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.613566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a11148e-837d-4b22-9b37-9b55b52b4714-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.619531 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.619509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzqfg\" (UniqueName: \"kubernetes.io/projected/3a11148e-837d-4b22-9b37-9b55b52b4714-kube-api-access-fzqfg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m495g\" (UID: \"3a11148e-837d-4b22-9b37-9b55b52b4714\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.758184 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.758162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:43.890855 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.890735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"]
Jul 24 17:16:43.893498 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:16:43.893476    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a11148e_837d_4b22_9b37_9b55b52b4714.slice/crio-c75c74febe114a86225e0e806e2acb64cc179d83052f85b4380d450d0ac2da3a WatchSource:0}: Error finding container c75c74febe114a86225e0e806e2acb64cc179d83052f85b4380d450d0ac2da3a: Status 404 returned error can't find the container with id c75c74febe114a86225e0e806e2acb64cc179d83052f85b4380d450d0ac2da3a
Jul 24 17:16:43.910928 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:43.910894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g" event={"ID":"3a11148e-837d-4b22-9b37-9b55b52b4714","Type":"ContainerStarted","Data":"c75c74febe114a86225e0e806e2acb64cc179d83052f85b4380d450d0ac2da3a"}
Jul 24 17:16:44.916895 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:44.916858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g" event={"ID":"3a11148e-837d-4b22-9b37-9b55b52b4714","Type":"ContainerStarted","Data":"a98acc759a01b5414ce5f14534ec6b830852ebe23118155b1886b3e516a2b756"}
Jul 24 17:16:44.937486 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:44.937441    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g" podStartSLOduration=1.7498671639999999 podStartE2EDuration="1.937425159s" totalImagesPullingTime="187.557995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:16:44.935202697 +0000 UTC m=+808.766455384" watchObservedRunningTime="2026-07-24 17:16:44.937425159 +0000 UTC m=+808.768677845"
Jul 24 17:16:51.786106 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:51.786075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:53.758393 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:53.758359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:53.758873 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:53.758476    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:53.771530 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:53.771503    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:16:53.973060 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:16:53.973033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m495g"
Jul 24 17:17:04.653825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:17:04.653755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:12.801050 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:17:12.801020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:42.227317 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:17:42.227262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:15.410586 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:15.410499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:15.508057 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:15.508031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:16.607444 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:16.607417    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:18:16.607866 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:16.607534    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:18:16.680801 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:16.680769    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:16.775143 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:16.775112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:40.604512 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:40.604476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:41.821833 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:41.821802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:53.889365 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:53.889331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:53.943668 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:18:53.943617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:09.216059 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.216025    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-87594f4b5-nfcg4"]
Jul 24 17:19:09.216446 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.216189    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-87594f4b5-nfcg4" podUID="5978becc-dbce-465f-8efc-d3f39dbb3ccc" containerName="manager" containerID="cri-o://0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2" gracePeriod=30
Jul 24 17:19:09.469773 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.469711    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:19:09.557467 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.557442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5978becc-dbce-465f-8efc-d3f39dbb3ccc-webhook-cert\") pod \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\" (UID: \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\") "
Jul 24 17:19:09.557623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.557487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwbrr\" (UniqueName: \"kubernetes.io/projected/5978becc-dbce-465f-8efc-d3f39dbb3ccc-kube-api-access-hwbrr\") pod \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\" (UID: \"5978becc-dbce-465f-8efc-d3f39dbb3ccc\") "
Jul 24 17:19:09.559581 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.559551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5978becc-dbce-465f-8efc-d3f39dbb3ccc-kube-api-access-hwbrr" (OuterVolumeSpecName: "kube-api-access-hwbrr") pod "5978becc-dbce-465f-8efc-d3f39dbb3ccc" (UID: "5978becc-dbce-465f-8efc-d3f39dbb3ccc"). InnerVolumeSpecName "kube-api-access-hwbrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:19:09.559581 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.559560    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5978becc-dbce-465f-8efc-d3f39dbb3ccc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5978becc-dbce-465f-8efc-d3f39dbb3ccc" (UID: "5978becc-dbce-465f-8efc-d3f39dbb3ccc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:19:09.572264 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.572238    2563 generic.go:356] "Generic (PLEG): container finished" podID="5978becc-dbce-465f-8efc-d3f39dbb3ccc" containerID="0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2" exitCode=0
Jul 24 17:19:09.572367 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.572280    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-nfcg4"
Jul 24 17:19:09.572367 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.572292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-nfcg4" event={"ID":"5978becc-dbce-465f-8efc-d3f39dbb3ccc","Type":"ContainerDied","Data":"0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2"}
Jul 24 17:19:09.572367 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.572355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-nfcg4" event={"ID":"5978becc-dbce-465f-8efc-d3f39dbb3ccc","Type":"ContainerDied","Data":"ac1eef67c03082c40bfa16e0963b2909e13fec97c70cef76cfda4f153df97674"}
Jul 24 17:19:09.572485 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.572376    2563 scope.go:117] "RemoveContainer" containerID="0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2"
Jul 24 17:19:09.581173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.581159    2563 scope.go:117] "RemoveContainer" containerID="0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2"
Jul 24 17:19:09.581443 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:19:09.581425    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2\": container with ID starting with 0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2 not found: ID does not exist" containerID="0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2"
Jul 24 17:19:09.581500 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.581451    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2"} err="failed to get container status \"0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2\": rpc error: code = NotFound desc = could not find container \"0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2\": container with ID starting with 0482400f3350b050d67da8c3b3ee2096bced0ccad577bc26cd527dbbde319cc2 not found: ID does not exist"
Jul 24 17:19:09.592997 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.592973    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-87594f4b5-nfcg4"]
Jul 24 17:19:09.596464 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.596444    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-87594f4b5-nfcg4"]
Jul 24 17:19:09.658109 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.658089    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5978becc-dbce-465f-8efc-d3f39dbb3ccc-webhook-cert\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:19:09.658109 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:09.658106    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwbrr\" (UniqueName: \"kubernetes.io/projected/5978becc-dbce-465f-8efc-d3f39dbb3ccc-kube-api-access-hwbrr\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:19:10.689095 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:10.689065    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5978becc-dbce-465f-8efc-d3f39dbb3ccc" path="/var/lib/kubelet/pods/5978becc-dbce-465f-8efc-d3f39dbb3ccc/volumes"
Jul 24 17:19:40.097065 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:19:40.096988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:10.489825 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.489790    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-87594f4b5-tkchz"]
Jul 24 17:20:10.490263 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.490117    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5978becc-dbce-465f-8efc-d3f39dbb3ccc" containerName="manager"
Jul 24 17:20:10.490263 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.490128    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5978becc-dbce-465f-8efc-d3f39dbb3ccc" containerName="manager"
Jul 24 17:20:10.490263 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.490217    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5978becc-dbce-465f-8efc-d3f39dbb3ccc" containerName="manager"
Jul 24 17:20:10.493251 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.493235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:10.495720 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.495698    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:20:10.496299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.496285    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:20:10.496386 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.496311    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5lmsg\""
Jul 24 17:20:10.502682 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.502640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-87594f4b5-tkchz"]
Jul 24 17:20:10.539391 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.539370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-webhook-cert\") pod \"maas-controller-87594f4b5-tkchz\" (UID: \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\") " pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:10.539478 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.539410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vztfk\" (UniqueName: \"kubernetes.io/projected/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-kube-api-access-vztfk\") pod \"maas-controller-87594f4b5-tkchz\" (UID: \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\") " pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:10.640104 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.640080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vztfk\" (UniqueName: \"kubernetes.io/projected/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-kube-api-access-vztfk\") pod \"maas-controller-87594f4b5-tkchz\" (UID: \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\") " pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:10.640194 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.640149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-webhook-cert\") pod \"maas-controller-87594f4b5-tkchz\" (UID: \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\") " pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:10.642712 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.642692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-webhook-cert\") pod \"maas-controller-87594f4b5-tkchz\" (UID: \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\") " pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:10.649338 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.649319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vztfk\" (UniqueName: \"kubernetes.io/projected/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-kube-api-access-vztfk\") pod \"maas-controller-87594f4b5-tkchz\" (UID: \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\") " pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:10.803640 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.803584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:10.937604 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.937494    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-87594f4b5-tkchz"]
Jul 24 17:20:10.940259 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:20:10.940229    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b365d6f_c95c_445b_8dbc_fb88d2d5b8b9.slice/crio-2aa8aa474470f51a99440d060f9e33516c8f97406655b6abaf1b1e63d9d040ab WatchSource:0}: Error finding container 2aa8aa474470f51a99440d060f9e33516c8f97406655b6abaf1b1e63d9d040ab: Status 404 returned error can't find the container with id 2aa8aa474470f51a99440d060f9e33516c8f97406655b6abaf1b1e63d9d040ab
Jul 24 17:20:10.941680 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:10.941639    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:20:11.866175 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:11.866094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-tkchz" event={"ID":"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9","Type":"ContainerStarted","Data":"2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0"}
Jul 24 17:20:11.866175 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:11.866142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-tkchz" event={"ID":"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9","Type":"ContainerStarted","Data":"2aa8aa474470f51a99440d060f9e33516c8f97406655b6abaf1b1e63d9d040ab"}
Jul 24 17:20:11.866546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:11.866280    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:11.886874 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:11.886830    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-87594f4b5-tkchz" podStartSLOduration=1.259088931 podStartE2EDuration="1.886817147s" totalImagesPullingTime="627.728216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:11.885994214 +0000 UTC m=+1015.717246903" watchObservedRunningTime="2026-07-24 17:20:11.886817147 +0000 UTC m=+1015.718069833"
Jul 24 17:20:22.874119 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:22.874080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:27.447168 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.447133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:27.560296 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.560261    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-87594f4b5-tkchz"]
Jul 24 17:20:27.560453 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.560433    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-87594f4b5-tkchz" podUID="3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9" containerName="manager" containerID="cri-o://2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0" gracePeriod=30
Jul 24 17:20:27.804895 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.804872    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:27.866411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.866386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-webhook-cert\") pod \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\" (UID: \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\") "
Jul 24 17:20:27.866550 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.866506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vztfk\" (UniqueName: \"kubernetes.io/projected/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-kube-api-access-vztfk\") pod \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\" (UID: \"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9\") "
Jul 24 17:20:27.868537 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.868514    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-kube-api-access-vztfk" (OuterVolumeSpecName: "kube-api-access-vztfk") pod "3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9" (UID: "3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9"). InnerVolumeSpecName "kube-api-access-vztfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:20:27.868660 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.868593    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9" (UID: "3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:20:27.944615 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.944588    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9" containerID="2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0" exitCode=0
Jul 24 17:20:27.944751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.944634    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-tkchz"
Jul 24 17:20:27.944751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.944672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-tkchz" event={"ID":"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9","Type":"ContainerDied","Data":"2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0"}
Jul 24 17:20:27.944859 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.944753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-tkchz" event={"ID":"3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9","Type":"ContainerDied","Data":"2aa8aa474470f51a99440d060f9e33516c8f97406655b6abaf1b1e63d9d040ab"}
Jul 24 17:20:27.944859 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.944787    2563 scope.go:117] "RemoveContainer" containerID="2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0"
Jul 24 17:20:27.955467 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.955444    2563 scope.go:117] "RemoveContainer" containerID="2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0"
Jul 24 17:20:27.955978 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:20:27.955951    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0\": container with ID starting with 2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0 not found: ID does not exist" containerID="2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0"
Jul 24 17:20:27.956068 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.955983    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0"} err="failed to get container status \"2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0\": rpc error: code = NotFound desc = could not find container \"2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0\": container with ID starting with 2c13385930e34d48e88c578525176c12a2f3b1fa324a275ec2e060c42411aee0 not found: ID does not exist"
Jul 24 17:20:27.964785 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.964763    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-87594f4b5-tkchz"]
Jul 24 17:20:27.967194 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.967176    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vztfk\" (UniqueName: \"kubernetes.io/projected/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-kube-api-access-vztfk\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:20:27.967248 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.967196    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9-webhook-cert\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:20:27.970643 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:27.970624    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-87594f4b5-tkchz"]
Jul 24 17:20:28.688235 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.688206    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9" path="/var/lib/kubelet/pods/3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9/volumes"
Jul 24 17:20:28.719178 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.719155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:28.838433 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.838411    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-87594f4b5-wfjwj"]
Jul 24 17:20:28.838771 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.838758    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9" containerName="manager"
Jul 24 17:20:28.838819 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.838773    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9" containerName="manager"
Jul 24 17:20:28.838853 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.838834    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b365d6f-c95c-445b-8dbc-fb88d2d5b8b9" containerName="manager"
Jul 24 17:20:28.843272 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.843255    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:28.845511 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.845494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5lmsg\""
Jul 24 17:20:28.846151 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.846137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:20:28.846254 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.846178    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:20:28.854308 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.854285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-87594f4b5-wfjwj"]
Jul 24 17:20:28.874493 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.874425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs522\" (UniqueName: \"kubernetes.io/projected/b7530b11-63ea-47af-a4f9-abcb2150c240-kube-api-access-cs522\") pod \"maas-controller-87594f4b5-wfjwj\" (UID: \"b7530b11-63ea-47af-a4f9-abcb2150c240\") " pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:28.874493 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.874478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7530b11-63ea-47af-a4f9-abcb2150c240-webhook-cert\") pod \"maas-controller-87594f4b5-wfjwj\" (UID: \"b7530b11-63ea-47af-a4f9-abcb2150c240\") " pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:28.975326 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.975304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7530b11-63ea-47af-a4f9-abcb2150c240-webhook-cert\") pod \"maas-controller-87594f4b5-wfjwj\" (UID: \"b7530b11-63ea-47af-a4f9-abcb2150c240\") " pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:28.975402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.975357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs522\" (UniqueName: \"kubernetes.io/projected/b7530b11-63ea-47af-a4f9-abcb2150c240-kube-api-access-cs522\") pod \"maas-controller-87594f4b5-wfjwj\" (UID: \"b7530b11-63ea-47af-a4f9-abcb2150c240\") " pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:28.977941 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.977921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7530b11-63ea-47af-a4f9-abcb2150c240-webhook-cert\") pod \"maas-controller-87594f4b5-wfjwj\" (UID: \"b7530b11-63ea-47af-a4f9-abcb2150c240\") " pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:28.984284 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:28.984262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs522\" (UniqueName: \"kubernetes.io/projected/b7530b11-63ea-47af-a4f9-abcb2150c240-kube-api-access-cs522\") pod \"maas-controller-87594f4b5-wfjwj\" (UID: \"b7530b11-63ea-47af-a4f9-abcb2150c240\") " pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:29.153809 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:29.153759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:29.280450 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:29.280316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-87594f4b5-wfjwj"]
Jul 24 17:20:29.283318 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:20:29.283293    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7530b11_63ea_47af_a4f9_abcb2150c240.slice/crio-76048cd786596c6cae21afedfeed877a10771172904c8ef1abb81fe5f0393372 WatchSource:0}: Error finding container 76048cd786596c6cae21afedfeed877a10771172904c8ef1abb81fe5f0393372: Status 404 returned error can't find the container with id 76048cd786596c6cae21afedfeed877a10771172904c8ef1abb81fe5f0393372
Jul 24 17:20:29.956310 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:29.956280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-wfjwj" event={"ID":"b7530b11-63ea-47af-a4f9-abcb2150c240","Type":"ContainerStarted","Data":"e6784148ea46138d9190470133262ed08027eaf9856cd65dfe82a7c6720bf357"}
Jul 24 17:20:29.956717 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:29.956332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-87594f4b5-wfjwj" event={"ID":"b7530b11-63ea-47af-a4f9-abcb2150c240","Type":"ContainerStarted","Data":"76048cd786596c6cae21afedfeed877a10771172904c8ef1abb81fe5f0393372"}
Jul 24 17:20:29.956717 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:29.956387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:29.976325 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:29.976244    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-87594f4b5-wfjwj" podStartSLOduration=1.5085329600000001 podStartE2EDuration="1.976227337s" totalImagesPullingTime="467.694377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:20:29.976071471 +0000 UTC m=+1033.807324159" watchObservedRunningTime="2026-07-24 17:20:29.976227337 +0000 UTC m=+1033.807480028"
Jul 24 17:20:40.963101 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:40.963068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-87594f4b5-wfjwj"
Jul 24 17:20:42.471199 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:42.471164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:42.538944 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:42.538918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:58.734082 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:20:58.734047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:07.412787 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:21:07.412700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:07.514796 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:21:07.514766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:07.666413 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:21:07.666332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:13.773409 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:21:13.773374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:13.881687 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:21:13.881633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:13.973603 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:21:13.973575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:14.134696 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:21:14.134603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:19.927633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:21:19.927600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:20.004322 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:21:20.004289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:23:16.643867 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:23:16.643801    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:23:16.644361 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:23:16.644311    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:26:40.834828 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:26:40.834799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:42.255388 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:26:42.255355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:28:16.672893 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:28:16.672867    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:28:16.674502 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:28:16.674471    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:28:42.557605 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:28:42.557528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:29:07.168117 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.168086    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"]
Jul 24 17:29:07.171750 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.171727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.174182 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.174158    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8c26ce9e-kserve-self-signed-certs\""
Jul 24 17:29:07.188998 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.188973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"]
Jul 24 17:29:07.194853 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.194834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:29:07.296830 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.296795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-model-cache\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.297004 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.296832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-dshm\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.297004 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.296874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-tmp-dir\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.297004 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.296905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd7qb\" (UniqueName: \"kubernetes.io/projected/0538c45b-6217-4ed1-a5f3-91da6f053032-kube-api-access-fd7qb\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.297004 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.296967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-home\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.297004 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.296997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0538c45b-6217-4ed1-a5f3-91da6f053032-tls-certs\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.398071 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.398038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-model-cache\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.398071 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.398076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-dshm\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.398287 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.398103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-tmp-dir\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.398287 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.398125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd7qb\" (UniqueName: \"kubernetes.io/projected/0538c45b-6217-4ed1-a5f3-91da6f053032-kube-api-access-fd7qb\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.398287 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.398155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-home\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.398287 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.398187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0538c45b-6217-4ed1-a5f3-91da6f053032-tls-certs\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.398489 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.398420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-model-cache\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.398532 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.398511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-tmp-dir\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.398619 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.398596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-home\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.400585 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.400560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-dshm\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.400718 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.400618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0538c45b-6217-4ed1-a5f3-91da6f053032-tls-certs\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.414344 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.414326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd7qb\" (UniqueName: \"kubernetes.io/projected/0538c45b-6217-4ed1-a5f3-91da6f053032-kube-api-access-fd7qb\") pod \"e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") " pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.480666 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.480624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:07.614908 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:07.614879    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"]
Jul 24 17:29:07.616900 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:29:07.616874    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0538c45b_6217_4ed1_a5f3_91da6f053032.slice/crio-af3f9c8e182f4824080b0b36047693313d3d15f082a9207ab09152a1da7b03f2 WatchSource:0}: Error finding container af3f9c8e182f4824080b0b36047693313d3d15f082a9207ab09152a1da7b03f2: Status 404 returned error can't find the container with id af3f9c8e182f4824080b0b36047693313d3d15f082a9207ab09152a1da7b03f2
Jul 24 17:29:08.254765 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:08.254732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt" event={"ID":"0538c45b-6217-4ed1-a5f3-91da6f053032","Type":"ContainerStarted","Data":"e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99"}
Jul 24 17:29:08.255136 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:08.254774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt" event={"ID":"0538c45b-6217-4ed1-a5f3-91da6f053032","Type":"ContainerStarted","Data":"af3f9c8e182f4824080b0b36047693313d3d15f082a9207ab09152a1da7b03f2"}
Jul 24 17:29:08.273551 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:08.273504    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt" podStartSLOduration=1.273489396 podStartE2EDuration="1.273489396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:08.27254035 +0000 UTC m=+1552.103793037" watchObservedRunningTime="2026-07-24 17:29:08.273489396 +0000 UTC m=+1552.104742083"
Jul 24 17:29:13.518505 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:13.518477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:29:17.481222 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:17.481190    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:17.481569 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:17.481239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:17.494191 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:17.494168    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:18.310535 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:18.310498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:32.533447 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:32.533417    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"]
Jul 24 17:29:32.568275 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:32.568227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:29:34.000154 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:34.000127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:29:47.552918 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.552865    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt" podUID="0538c45b-6217-4ed1-a5f3-91da6f053032" containerName="main" containerID="cri-o://e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99" gracePeriod=45
Jul 24 17:29:47.798413 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.798385    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:47.935419 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.935345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0538c45b-6217-4ed1-a5f3-91da6f053032-tls-certs\") pod \"0538c45b-6217-4ed1-a5f3-91da6f053032\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") "
Jul 24 17:29:47.935419 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.935392    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-dshm\") pod \"0538c45b-6217-4ed1-a5f3-91da6f053032\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") "
Jul 24 17:29:47.935631 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.935443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd7qb\" (UniqueName: \"kubernetes.io/projected/0538c45b-6217-4ed1-a5f3-91da6f053032-kube-api-access-fd7qb\") pod \"0538c45b-6217-4ed1-a5f3-91da6f053032\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") "
Jul 24 17:29:47.935631 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.935485    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-tmp-dir\") pod \"0538c45b-6217-4ed1-a5f3-91da6f053032\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") "
Jul 24 17:29:47.935631 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.935506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-home\") pod \"0538c45b-6217-4ed1-a5f3-91da6f053032\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") "
Jul 24 17:29:47.935631 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.935558    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-model-cache\") pod \"0538c45b-6217-4ed1-a5f3-91da6f053032\" (UID: \"0538c45b-6217-4ed1-a5f3-91da6f053032\") "
Jul 24 17:29:47.935850 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.935780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0538c45b-6217-4ed1-a5f3-91da6f053032" (UID: "0538c45b-6217-4ed1-a5f3-91da6f053032"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:29:47.935910 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.935861    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-home" (OuterVolumeSpecName: "home") pod "0538c45b-6217-4ed1-a5f3-91da6f053032" (UID: "0538c45b-6217-4ed1-a5f3-91da6f053032"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:29:47.935910 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.935897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-model-cache" (OuterVolumeSpecName: "model-cache") pod "0538c45b-6217-4ed1-a5f3-91da6f053032" (UID: "0538c45b-6217-4ed1-a5f3-91da6f053032"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:29:47.937722 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.937694    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-dshm" (OuterVolumeSpecName: "dshm") pod "0538c45b-6217-4ed1-a5f3-91da6f053032" (UID: "0538c45b-6217-4ed1-a5f3-91da6f053032"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:29:47.937841 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.937798    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0538c45b-6217-4ed1-a5f3-91da6f053032-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0538c45b-6217-4ed1-a5f3-91da6f053032" (UID: "0538c45b-6217-4ed1-a5f3-91da6f053032"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:29:47.937893 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:47.937864    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0538c45b-6217-4ed1-a5f3-91da6f053032-kube-api-access-fd7qb" (OuterVolumeSpecName: "kube-api-access-fd7qb") pod "0538c45b-6217-4ed1-a5f3-91da6f053032" (UID: "0538c45b-6217-4ed1-a5f3-91da6f053032"). InnerVolumeSpecName "kube-api-access-fd7qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:29:48.037062 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.037029    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0538c45b-6217-4ed1-a5f3-91da6f053032-tls-certs\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:29:48.037062 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.037057    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-dshm\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:29:48.037214 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.037070    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd7qb\" (UniqueName: \"kubernetes.io/projected/0538c45b-6217-4ed1-a5f3-91da6f053032-kube-api-access-fd7qb\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:29:48.037214 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.037081    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-tmp-dir\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:29:48.037214 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.037092    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-home\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:29:48.037214 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.037102    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0538c45b-6217-4ed1-a5f3-91da6f053032-model-cache\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:29:48.431364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.431330    2563 generic.go:356] "Generic (PLEG): container finished" podID="0538c45b-6217-4ed1-a5f3-91da6f053032" containerID="e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99" exitCode=0
Jul 24 17:29:48.431495 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.431369    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"
Jul 24 17:29:48.431495 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.431390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt" event={"ID":"0538c45b-6217-4ed1-a5f3-91da6f053032","Type":"ContainerDied","Data":"e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99"}
Jul 24 17:29:48.431495 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.431431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt" event={"ID":"0538c45b-6217-4ed1-a5f3-91da6f053032","Type":"ContainerDied","Data":"af3f9c8e182f4824080b0b36047693313d3d15f082a9207ab09152a1da7b03f2"}
Jul 24 17:29:48.431495 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.431449    2563 scope.go:117] "RemoveContainer" containerID="e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99"
Jul 24 17:29:48.440299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.440281    2563 scope.go:117] "RemoveContainer" containerID="e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99"
Jul 24 17:29:48.440543 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:29:48.440523    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99\": container with ID starting with e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99 not found: ID does not exist" containerID="e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99"
Jul 24 17:29:48.440675 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.440555    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99"} err="failed to get container status \"e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99\": rpc error: code = NotFound desc = could not find container \"e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99\": container with ID starting with e5c7db6761c614da1621ad1c36d7d729e476bbb8e122acaee0cfe5d0341aea99 not found: ID does not exist"
Jul 24 17:29:48.451922 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.451892    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"]
Jul 24 17:29:48.457524 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.457505    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8c26ce9e-kserve-6c8449895f-mrlqt"]
Jul 24 17:29:48.688673 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:29:48.688581    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0538c45b-6217-4ed1-a5f3-91da6f053032" path="/var/lib/kubelet/pods/0538c45b-6217-4ed1-a5f3-91da6f053032/volumes"
Jul 24 17:31:50.197084 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:31:50.197004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:31:51.491213 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:31:51.491151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:32:45.716170 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:32:45.716128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:02.826281 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:02.826240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:07.210594 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:07.210524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:10.294236 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:10.294179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:10.390145 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:10.390119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:10.731900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:10.731857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:16.702354 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:16.702330    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:33:16.705623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:16.705597    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:33:16.960400 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:16.960330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:17.129749 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:17.129725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:17.231034 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:17.231009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:17.724245 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:17.724220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:18.196127 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:18.196027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:19.881093 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:19.881064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:19.986058 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:19.986000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:20.102855 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.102824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:20.295956 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.295923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:20.683782 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.683704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:20.787250 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.787218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:20.834616 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.834586    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"]
Jul 24 17:33:20.834984 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.834971    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0538c45b-6217-4ed1-a5f3-91da6f053032" containerName="main"
Jul 24 17:33:20.835026 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.834985    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0538c45b-6217-4ed1-a5f3-91da6f053032" containerName="main"
Jul 24 17:33:20.835079 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.835070    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0538c45b-6217-4ed1-a5f3-91da6f053032" containerName="main"
Jul 24 17:33:20.837997 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.837982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:20.844045 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.844025    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f231d414\""
Jul 24 17:33:20.850602 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.850582    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"]
Jul 24 17:33:20.890200 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.890175    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"]
Jul 24 17:33:20.893331 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.893317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:20.896587 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.896567    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f231d414-dockercfg-c4nw9\""
Jul 24 17:33:20.910591 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.910567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"]
Jul 24 17:33:20.927501 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.927480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92fq5\" (UniqueName: \"kubernetes.io/projected/46113e0b-f0d5-4963-b30b-23e1a73daf1a-kube-api-access-92fq5\") pod \"payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw\" (UID: \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\") " pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:20.927604 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:20.927525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46113e0b-f0d5-4963-b30b-23e1a73daf1a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw\" (UID: \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\") " pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:21.028357 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.028328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92fq5\" (UniqueName: \"kubernetes.io/projected/46113e0b-f0d5-4963-b30b-23e1a73daf1a-kube-api-access-92fq5\") pod \"payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw\" (UID: \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\") " pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:21.028505 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.028377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbcb\" (UniqueName: \"kubernetes.io/projected/9a7ed77e-2d1f-4877-b243-30113e491d8c-kube-api-access-wdbcb\") pod \"payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk\" (UID: \"9a7ed77e-2d1f-4877-b243-30113e491d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:21.028505 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.028412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46113e0b-f0d5-4963-b30b-23e1a73daf1a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw\" (UID: \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\") " pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:21.028505 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.028487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7ed77e-2d1f-4877-b243-30113e491d8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk\" (UID: \"9a7ed77e-2d1f-4877-b243-30113e491d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:21.029075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.029058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46113e0b-f0d5-4963-b30b-23e1a73daf1a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw\" (UID: \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\") " pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:21.038690 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.038669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92fq5\" (UniqueName: \"kubernetes.io/projected/46113e0b-f0d5-4963-b30b-23e1a73daf1a-kube-api-access-92fq5\") pod \"payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw\" (UID: \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\") " pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:21.129356 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.129325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbcb\" (UniqueName: \"kubernetes.io/projected/9a7ed77e-2d1f-4877-b243-30113e491d8c-kube-api-access-wdbcb\") pod \"payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk\" (UID: \"9a7ed77e-2d1f-4877-b243-30113e491d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:21.129480 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.129398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7ed77e-2d1f-4877-b243-30113e491d8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk\" (UID: \"9a7ed77e-2d1f-4877-b243-30113e491d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:21.130048 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.130031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7ed77e-2d1f-4877-b243-30113e491d8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk\" (UID: \"9a7ed77e-2d1f-4877-b243-30113e491d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:21.137921 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.137902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbcb\" (UniqueName: \"kubernetes.io/projected/9a7ed77e-2d1f-4877-b243-30113e491d8c-kube-api-access-wdbcb\") pod \"payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk\" (UID: \"9a7ed77e-2d1f-4877-b243-30113e491d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:21.148425 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.148407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:21.202635 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.202604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:21.337555 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.337528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"]
Jul 24 17:33:21.340002 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:33:21.339973    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7ed77e_2d1f_4877_b243_30113e491d8c.slice/crio-8d4295e050e44025d721ccd1893aaa75ee22bd18c62390b3483d50709c952785 WatchSource:0}: Error finding container 8d4295e050e44025d721ccd1893aaa75ee22bd18c62390b3483d50709c952785: Status 404 returned error can't find the container with id 8d4295e050e44025d721ccd1893aaa75ee22bd18c62390b3483d50709c952785
Jul 24 17:33:21.385360 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.385335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk" event={"ID":"9a7ed77e-2d1f-4877-b243-30113e491d8c","Type":"ContainerStarted","Data":"8d4295e050e44025d721ccd1893aaa75ee22bd18c62390b3483d50709c952785"}
Jul 24 17:33:21.492708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:21.492680    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"]
Jul 24 17:33:21.495662 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:33:21.495622    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46113e0b_f0d5_4963_b30b_23e1a73daf1a.slice/crio-f94071cfd85c6e604b646820157898c6207d80621058a5f4b66762a28581a160 WatchSource:0}: Error finding container f94071cfd85c6e604b646820157898c6207d80621058a5f4b66762a28581a160: Status 404 returned error can't find the container with id f94071cfd85c6e604b646820157898c6207d80621058a5f4b66762a28581a160
Jul 24 17:33:22.391665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:22.391623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw" event={"ID":"46113e0b-f0d5-4963-b30b-23e1a73daf1a","Type":"ContainerStarted","Data":"ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f"}
Jul 24 17:33:22.392135 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:22.391696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw" event={"ID":"46113e0b-f0d5-4963-b30b-23e1a73daf1a","Type":"ContainerStarted","Data":"f94071cfd85c6e604b646820157898c6207d80621058a5f4b66762a28581a160"}
Jul 24 17:33:22.392135 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:22.391724    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:22.393060 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:22.393033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk" event={"ID":"9a7ed77e-2d1f-4877-b243-30113e491d8c","Type":"ContainerStarted","Data":"60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24"}
Jul 24 17:33:22.393178 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:22.393159    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:22.407578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:22.407531    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw" podStartSLOduration=2.40751719 podStartE2EDuration="2.40751719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:33:22.405741168 +0000 UTC m=+1806.236993856" watchObservedRunningTime="2026-07-24 17:33:22.40751719 +0000 UTC m=+1806.238769875"
Jul 24 17:33:22.421441 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:22.421403    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk" podStartSLOduration=2.421391363 podStartE2EDuration="2.421391363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:33:22.420448233 +0000 UTC m=+1806.251700921" watchObservedRunningTime="2026-07-24 17:33:22.421391363 +0000 UTC m=+1806.252644047"
Jul 24 17:33:33.399780 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:33.399749    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:33.400295 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:33.399824    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:55.826982 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:55.826949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:55.863785 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:55.863754    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"]
Jul 24 17:33:55.864046 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:55.864019    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw" podUID="46113e0b-f0d5-4963-b30b-23e1a73daf1a" containerName="payload-processing" containerID="cri-o://ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f" gracePeriod=30
Jul 24 17:33:55.871511 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:55.871484    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"]
Jul 24 17:33:55.871763 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:55.871719    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk" podUID="9a7ed77e-2d1f-4877-b243-30113e491d8c" containerName="payload-pre-processing" containerID="cri-o://60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24" gracePeriod=30
Jul 24 17:33:55.910479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:55.910450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:55.980746 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:55.980724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:56.139544 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.139519    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:56.142320 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.142300    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:56.213310 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.213283    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92fq5\" (UniqueName: \"kubernetes.io/projected/46113e0b-f0d5-4963-b30b-23e1a73daf1a-kube-api-access-92fq5\") pod \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\" (UID: \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\") "
Jul 24 17:33:56.213497 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.213349    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46113e0b-f0d5-4963-b30b-23e1a73daf1a-plugins-config-volume\") pod \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\" (UID: \"46113e0b-f0d5-4963-b30b-23e1a73daf1a\") "
Jul 24 17:33:56.213497 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.213372    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdbcb\" (UniqueName: \"kubernetes.io/projected/9a7ed77e-2d1f-4877-b243-30113e491d8c-kube-api-access-wdbcb\") pod \"9a7ed77e-2d1f-4877-b243-30113e491d8c\" (UID: \"9a7ed77e-2d1f-4877-b243-30113e491d8c\") "
Jul 24 17:33:56.213497 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.213408    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7ed77e-2d1f-4877-b243-30113e491d8c-plugins-config-volume\") pod \"9a7ed77e-2d1f-4877-b243-30113e491d8c\" (UID: \"9a7ed77e-2d1f-4877-b243-30113e491d8c\") "
Jul 24 17:33:56.213798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.213772    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46113e0b-f0d5-4963-b30b-23e1a73daf1a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46113e0b-f0d5-4963-b30b-23e1a73daf1a" (UID: "46113e0b-f0d5-4963-b30b-23e1a73daf1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:33:56.213880 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.213807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7ed77e-2d1f-4877-b243-30113e491d8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a7ed77e-2d1f-4877-b243-30113e491d8c" (UID: "9a7ed77e-2d1f-4877-b243-30113e491d8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:33:56.215631 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.215609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7ed77e-2d1f-4877-b243-30113e491d8c-kube-api-access-wdbcb" (OuterVolumeSpecName: "kube-api-access-wdbcb") pod "9a7ed77e-2d1f-4877-b243-30113e491d8c" (UID: "9a7ed77e-2d1f-4877-b243-30113e491d8c"). InnerVolumeSpecName "kube-api-access-wdbcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:33:56.215743 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.215724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46113e0b-f0d5-4963-b30b-23e1a73daf1a-kube-api-access-92fq5" (OuterVolumeSpecName: "kube-api-access-92fq5") pod "46113e0b-f0d5-4963-b30b-23e1a73daf1a" (UID: "46113e0b-f0d5-4963-b30b-23e1a73daf1a"). InnerVolumeSpecName "kube-api-access-92fq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:33:56.315007 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.314975    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46113e0b-f0d5-4963-b30b-23e1a73daf1a-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:33:56.315007 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.315002    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbcb\" (UniqueName: \"kubernetes.io/projected/9a7ed77e-2d1f-4877-b243-30113e491d8c-kube-api-access-wdbcb\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:33:56.315007 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.315012    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7ed77e-2d1f-4877-b243-30113e491d8c-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:33:56.315276 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.315022    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92fq5\" (UniqueName: \"kubernetes.io/projected/46113e0b-f0d5-4963-b30b-23e1a73daf1a-kube-api-access-92fq5\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:33:56.548539 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.548501    2563 generic.go:356] "Generic (PLEG): container finished" podID="46113e0b-f0d5-4963-b30b-23e1a73daf1a" containerID="ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f" exitCode=0
Jul 24 17:33:56.548728 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.548552    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"
Jul 24 17:33:56.548728 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.548586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw" event={"ID":"46113e0b-f0d5-4963-b30b-23e1a73daf1a","Type":"ContainerDied","Data":"ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f"}
Jul 24 17:33:56.548728 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.548642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw" event={"ID":"46113e0b-f0d5-4963-b30b-23e1a73daf1a","Type":"ContainerDied","Data":"f94071cfd85c6e604b646820157898c6207d80621058a5f4b66762a28581a160"}
Jul 24 17:33:56.548728 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.548692    2563 scope.go:117] "RemoveContainer" containerID="ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f"
Jul 24 17:33:56.550020 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.549995    2563 generic.go:356] "Generic (PLEG): container finished" podID="9a7ed77e-2d1f-4877-b243-30113e491d8c" containerID="60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24" exitCode=0
Jul 24 17:33:56.550200 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.550038    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"
Jul 24 17:33:56.550200 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.550053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk" event={"ID":"9a7ed77e-2d1f-4877-b243-30113e491d8c","Type":"ContainerDied","Data":"60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24"}
Jul 24 17:33:56.550200 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.550094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk" event={"ID":"9a7ed77e-2d1f-4877-b243-30113e491d8c","Type":"ContainerDied","Data":"8d4295e050e44025d721ccd1893aaa75ee22bd18c62390b3483d50709c952785"}
Jul 24 17:33:56.558170 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.558151    2563 scope.go:117] "RemoveContainer" containerID="ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f"
Jul 24 17:33:56.558426 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:33:56.558409    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f\": container with ID starting with ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f not found: ID does not exist" containerID="ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f"
Jul 24 17:33:56.558506 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.558431    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f"} err="failed to get container status \"ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f\": rpc error: code = NotFound desc = could not find container \"ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f\": container with ID starting with ac636c3029412e1506113bbcdab4bb9297e45cd4dedcec6ec64e7f1b1beeb54f not found: ID does not exist"
Jul 24 17:33:56.558506 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.558447    2563 scope.go:117] "RemoveContainer" containerID="60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24"
Jul 24 17:33:56.566261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.566246    2563 scope.go:117] "RemoveContainer" containerID="60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24"
Jul 24 17:33:56.566499 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:33:56.566482    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24\": container with ID starting with 60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24 not found: ID does not exist" containerID="60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24"
Jul 24 17:33:56.566540 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.566505    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24"} err="failed to get container status \"60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24\": rpc error: code = NotFound desc = could not find container \"60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24\": container with ID starting with 60758b098c59fd967f80e750468c62416b61e282a4f80e62493ce3bc32c39c24 not found: ID does not exist"
Jul 24 17:33:56.572382 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.572360    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"]
Jul 24 17:33:56.578839 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.578820    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f231d414-557748c7f8-zfrvw"]
Jul 24 17:33:56.588797 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.588767    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"]
Jul 24 17:33:56.590804 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.590787    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f231d414-5fdf8688cb-cmzbk"]
Jul 24 17:33:56.690288 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.690260    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46113e0b-f0d5-4963-b30b-23e1a73daf1a" path="/var/lib/kubelet/pods/46113e0b-f0d5-4963-b30b-23e1a73daf1a/volumes"
Jul 24 17:33:56.690642 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:33:56.690616    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a7ed77e-2d1f-4877-b243-30113e491d8c" path="/var/lib/kubelet/pods/9a7ed77e-2d1f-4877-b243-30113e491d8c/volumes"
Jul 24 17:34:00.442180 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:00.442092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:00.608820 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:00.608783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:00.850902 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:00.850864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:01.111324 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:01.111222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:01.226182 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:01.226117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:01.394051 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:01.393941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:01.671685 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:01.671526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:01.981025 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:01.980996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:04.268261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:04.268221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:04.459291 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:04.459254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:04.816250 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:04.816218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:05.032044 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.032004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:05.128291 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.128169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:05.149389 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.149357    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"]
Jul 24 17:34:05.149768 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.149753    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7ed77e-2d1f-4877-b243-30113e491d8c" containerName="payload-pre-processing"
Jul 24 17:34:05.149822 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.149770    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7ed77e-2d1f-4877-b243-30113e491d8c" containerName="payload-pre-processing"
Jul 24 17:34:05.149822 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.149782    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46113e0b-f0d5-4963-b30b-23e1a73daf1a" containerName="payload-processing"
Jul 24 17:34:05.149822 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.149788    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="46113e0b-f0d5-4963-b30b-23e1a73daf1a" containerName="payload-processing"
Jul 24 17:34:05.149916 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.149895    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="46113e0b-f0d5-4963-b30b-23e1a73daf1a" containerName="payload-processing"
Jul 24 17:34:05.149916 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.149906    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a7ed77e-2d1f-4877-b243-30113e491d8c" containerName="payload-pre-processing"
Jul 24 17:34:05.154557 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.154538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:05.157686 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.157641    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3f26bc98-dockercfg-5g9zj\""
Jul 24 17:34:05.157686 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.157673    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3f26bc98\""
Jul 24 17:34:05.163999 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.163978    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"]
Jul 24 17:34:05.170542 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.170519    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"]
Jul 24 17:34:05.174009 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.173981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:05.183116 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.183095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"]
Jul 24 17:34:05.295302 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.295258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5a0786-c075-4024-90a2-16ad4efad361-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns\" (UID: \"1d5a0786-c075-4024-90a2-16ad4efad361\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:05.295302 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.295291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgcmm\" (UniqueName: \"kubernetes.io/projected/1d5a0786-c075-4024-90a2-16ad4efad361-kube-api-access-qgcmm\") pod \"payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns\" (UID: \"1d5a0786-c075-4024-90a2-16ad4efad361\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:05.295842 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.295385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt\" (UID: \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:05.295842 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.295531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4h2w\" (UniqueName: \"kubernetes.io/projected/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-kube-api-access-k4h2w\") pod \"payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt\" (UID: \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:05.397125 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.397027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt\" (UID: \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:05.397125 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.397110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4h2w\" (UniqueName: \"kubernetes.io/projected/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-kube-api-access-k4h2w\") pod \"payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt\" (UID: \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:05.397308 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.397217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5a0786-c075-4024-90a2-16ad4efad361-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns\" (UID: \"1d5a0786-c075-4024-90a2-16ad4efad361\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:05.397308 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.397236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgcmm\" (UniqueName: \"kubernetes.io/projected/1d5a0786-c075-4024-90a2-16ad4efad361-kube-api-access-qgcmm\") pod \"payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns\" (UID: \"1d5a0786-c075-4024-90a2-16ad4efad361\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:05.397874 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.397856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt\" (UID: \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:05.398338 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.398317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5a0786-c075-4024-90a2-16ad4efad361-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns\" (UID: \"1d5a0786-c075-4024-90a2-16ad4efad361\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:05.405526 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.405502    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgcmm\" (UniqueName: \"kubernetes.io/projected/1d5a0786-c075-4024-90a2-16ad4efad361-kube-api-access-qgcmm\") pod \"payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns\" (UID: \"1d5a0786-c075-4024-90a2-16ad4efad361\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:05.405886 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.405865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4h2w\" (UniqueName: \"kubernetes.io/projected/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-kube-api-access-k4h2w\") pod \"payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt\" (UID: \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:05.468130 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.468075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:05.484835 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.484744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:05.625905 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.625859    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"]
Jul 24 17:34:05.629404 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:34:05.629369    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83b1310_a8a0_4eb0_a67d_76c6d0ca248d.slice/crio-d8675286afb800b45297ace3543d682ccab3818e6abe94e980f672db71abd420 WatchSource:0}: Error finding container d8675286afb800b45297ace3543d682ccab3818e6abe94e980f672db71abd420: Status 404 returned error can't find the container with id d8675286afb800b45297ace3543d682ccab3818e6abe94e980f672db71abd420
Jul 24 17:34:05.652484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:05.652457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"]
Jul 24 17:34:05.663802 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:34:05.663766    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5a0786_c075_4024_90a2_16ad4efad361.slice/crio-7add3de6fbd0800c3e293370b36db9c2160504255c4f94fa754fff0f7d36a441 WatchSource:0}: Error finding container 7add3de6fbd0800c3e293370b36db9c2160504255c4f94fa754fff0f7d36a441: Status 404 returned error can't find the container with id 7add3de6fbd0800c3e293370b36db9c2160504255c4f94fa754fff0f7d36a441
Jul 24 17:34:06.603211 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:06.603173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt" event={"ID":"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d","Type":"ContainerStarted","Data":"ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59"}
Jul 24 17:34:06.603718 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:06.603237    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:06.603718 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:06.603258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt" event={"ID":"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d","Type":"ContainerStarted","Data":"d8675286afb800b45297ace3543d682ccab3818e6abe94e980f672db71abd420"}
Jul 24 17:34:06.604911 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:06.604885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns" event={"ID":"1d5a0786-c075-4024-90a2-16ad4efad361","Type":"ContainerStarted","Data":"bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede"}
Jul 24 17:34:06.605037 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:06.604933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns" event={"ID":"1d5a0786-c075-4024-90a2-16ad4efad361","Type":"ContainerStarted","Data":"7add3de6fbd0800c3e293370b36db9c2160504255c4f94fa754fff0f7d36a441"}
Jul 24 17:34:06.605095 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:06.605030    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:06.620936 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:06.620887    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt" podStartSLOduration=1.620873394 podStartE2EDuration="1.620873394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:34:06.6201328 +0000 UTC m=+1850.451385487" watchObservedRunningTime="2026-07-24 17:34:06.620873394 +0000 UTC m=+1850.452126151"
Jul 24 17:34:06.636046 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:06.636001    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns" podStartSLOduration=1.6359845480000001 podStartE2EDuration="1.635984548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:34:06.635154205 +0000 UTC m=+1850.466406893" watchObservedRunningTime="2026-07-24 17:34:06.635984548 +0000 UTC m=+1850.467237305"
Jul 24 17:34:17.611149 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:17.611114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:17.611519 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:17.611200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:40.004325 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.003963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:40.007533 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.007506    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"]
Jul 24 17:34:40.007995 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.007759    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns" podUID="1d5a0786-c075-4024-90a2-16ad4efad361" containerName="payload-processing" containerID="cri-o://bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede" gracePeriod=30
Jul 24 17:34:40.039240 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.039214    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"]
Jul 24 17:34:40.039444 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.039409    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt" podUID="f83b1310-a8a0-4eb0-a67d-76c6d0ca248d" containerName="payload-pre-processing" containerID="cri-o://ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59" gracePeriod=30
Jul 24 17:34:40.092335 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.092311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:40.293605 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.293580    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:40.305329 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.305310    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:40.403150 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.403121    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-plugins-config-volume\") pod \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\" (UID: \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\") "
Jul 24 17:34:40.403341 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.403163    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4h2w\" (UniqueName: \"kubernetes.io/projected/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-kube-api-access-k4h2w\") pod \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\" (UID: \"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d\") "
Jul 24 17:34:40.403341 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.403262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgcmm\" (UniqueName: \"kubernetes.io/projected/1d5a0786-c075-4024-90a2-16ad4efad361-kube-api-access-qgcmm\") pod \"1d5a0786-c075-4024-90a2-16ad4efad361\" (UID: \"1d5a0786-c075-4024-90a2-16ad4efad361\") "
Jul 24 17:34:40.403341 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.403320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5a0786-c075-4024-90a2-16ad4efad361-plugins-config-volume\") pod \"1d5a0786-c075-4024-90a2-16ad4efad361\" (UID: \"1d5a0786-c075-4024-90a2-16ad4efad361\") "
Jul 24 17:34:40.403598 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.403553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f83b1310-a8a0-4eb0-a67d-76c6d0ca248d" (UID: "f83b1310-a8a0-4eb0-a67d-76c6d0ca248d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:34:40.403750 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.403697    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:34:40.404341 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.403928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5a0786-c075-4024-90a2-16ad4efad361-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d5a0786-c075-4024-90a2-16ad4efad361" (UID: "1d5a0786-c075-4024-90a2-16ad4efad361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:34:40.405475 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.405453    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-kube-api-access-k4h2w" (OuterVolumeSpecName: "kube-api-access-k4h2w") pod "f83b1310-a8a0-4eb0-a67d-76c6d0ca248d" (UID: "f83b1310-a8a0-4eb0-a67d-76c6d0ca248d"). InnerVolumeSpecName "kube-api-access-k4h2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:34:40.405567 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.405546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5a0786-c075-4024-90a2-16ad4efad361-kube-api-access-qgcmm" (OuterVolumeSpecName: "kube-api-access-qgcmm") pod "1d5a0786-c075-4024-90a2-16ad4efad361" (UID: "1d5a0786-c075-4024-90a2-16ad4efad361"). InnerVolumeSpecName "kube-api-access-qgcmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:34:40.504492 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.504458    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4h2w\" (UniqueName: \"kubernetes.io/projected/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d-kube-api-access-k4h2w\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:34:40.504492 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.504486    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgcmm\" (UniqueName: \"kubernetes.io/projected/1d5a0786-c075-4024-90a2-16ad4efad361-kube-api-access-qgcmm\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:34:40.504492 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.504499    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5a0786-c075-4024-90a2-16ad4efad361-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:34:40.758845 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.758815    2563 generic.go:356] "Generic (PLEG): container finished" podID="f83b1310-a8a0-4eb0-a67d-76c6d0ca248d" containerID="ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59" exitCode=0
Jul 24 17:34:40.759024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.758857    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"
Jul 24 17:34:40.759024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.758903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt" event={"ID":"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d","Type":"ContainerDied","Data":"ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59"}
Jul 24 17:34:40.759024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.758957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt" event={"ID":"f83b1310-a8a0-4eb0-a67d-76c6d0ca248d","Type":"ContainerDied","Data":"d8675286afb800b45297ace3543d682ccab3818e6abe94e980f672db71abd420"}
Jul 24 17:34:40.759024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.758989    2563 scope.go:117] "RemoveContainer" containerID="ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59"
Jul 24 17:34:40.760466 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.760448    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d5a0786-c075-4024-90a2-16ad4efad361" containerID="bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede" exitCode=0
Jul 24 17:34:40.760568 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.760476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns" event={"ID":"1d5a0786-c075-4024-90a2-16ad4efad361","Type":"ContainerDied","Data":"bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede"}
Jul 24 17:34:40.760568 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.760503    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"
Jul 24 17:34:40.760893 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.760505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns" event={"ID":"1d5a0786-c075-4024-90a2-16ad4efad361","Type":"ContainerDied","Data":"7add3de6fbd0800c3e293370b36db9c2160504255c4f94fa754fff0f7d36a441"}
Jul 24 17:34:40.767703 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.767688    2563 scope.go:117] "RemoveContainer" containerID="ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59"
Jul 24 17:34:40.767973 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:34:40.767955    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59\": container with ID starting with ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59 not found: ID does not exist" containerID="ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59"
Jul 24 17:34:40.768020 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.767982    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59"} err="failed to get container status \"ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59\": rpc error: code = NotFound desc = could not find container \"ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59\": container with ID starting with ab99bfd5ac7b8c5643b4559223a2ba7eab1517e3e75716fe4bfeeb175f408c59 not found: ID does not exist"
Jul 24 17:34:40.768020 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.767999    2563 scope.go:117] "RemoveContainer" containerID="bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede"
Jul 24 17:34:40.775587 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.775571    2563 scope.go:117] "RemoveContainer" containerID="bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede"
Jul 24 17:34:40.775926 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:34:40.775907    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede\": container with ID starting with bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede not found: ID does not exist" containerID="bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede"
Jul 24 17:34:40.775983 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.775933    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede"} err="failed to get container status \"bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede\": rpc error: code = NotFound desc = could not find container \"bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede\": container with ID starting with bb00a3b55ade2e8edef12ebe0d0f37722c6560819df01cc990dbe4a2695b1ede not found: ID does not exist"
Jul 24 17:34:40.792170 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.792140    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"]
Jul 24 17:34:40.803453 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.803432    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f26bc98-6fdd9969b9-79mns"]
Jul 24 17:34:40.835076 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.835055    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"]
Jul 24 17:34:40.847343 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:40.847320    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3f26bc98-78f5b67484-tcmnt"]
Jul 24 17:34:42.688368 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:42.688336    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5a0786-c075-4024-90a2-16ad4efad361" path="/var/lib/kubelet/pods/1d5a0786-c075-4024-90a2-16ad4efad361/volumes"
Jul 24 17:34:42.688750 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:42.688738    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f83b1310-a8a0-4eb0-a67d-76c6d0ca248d" path="/var/lib/kubelet/pods/f83b1310-a8a0-4eb0-a67d-76c6d0ca248d/volumes"
Jul 24 17:34:44.414496 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:44.414453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:44.569092 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:44.569062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:44.816580 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:44.816547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:45.104040 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:45.103963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:45.590058 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:45.590030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:45.693418 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:45.693385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:48.320895 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:48.320851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:48.428206 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:48.428159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:48.537124 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:48.537089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:48.808697 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:48.808642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:48.995350 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:48.995322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:49.102234 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.102146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:49.156415 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.156389    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"]
Jul 24 17:34:49.156777 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.156764    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5a0786-c075-4024-90a2-16ad4efad361" containerName="payload-processing"
Jul 24 17:34:49.156831 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.156779    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5a0786-c075-4024-90a2-16ad4efad361" containerName="payload-processing"
Jul 24 17:34:49.156831 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.156790    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83b1310-a8a0-4eb0-a67d-76c6d0ca248d" containerName="payload-pre-processing"
Jul 24 17:34:49.156831 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.156796    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83b1310-a8a0-4eb0-a67d-76c6d0ca248d" containerName="payload-pre-processing"
Jul 24 17:34:49.156927 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.156856    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5a0786-c075-4024-90a2-16ad4efad361" containerName="payload-processing"
Jul 24 17:34:49.156927 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.156863    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f83b1310-a8a0-4eb0-a67d-76c6d0ca248d" containerName="payload-pre-processing"
Jul 24 17:34:49.159768 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.159748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:34:49.163414 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.163399    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-74f5d728\""
Jul 24 17:34:49.163490 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.163427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-74f5d728-dockercfg-v6hfz\""
Jul 24 17:34:49.186582 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.186559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"]
Jul 24 17:34:49.202363 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.202335    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"]
Jul 24 17:34:49.205738 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.205722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:34:49.222979 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.222953    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"]
Jul 24 17:34:49.245669 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.245614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:49.282955 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.282917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzt5n\" (UniqueName: \"kubernetes.io/projected/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-kube-api-access-zzt5n\") pod \"payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5\" (UID: \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\") " pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:34:49.283125 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.282981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbd9t\" (UniqueName: \"kubernetes.io/projected/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-kube-api-access-jbd9t\") pod \"payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv\" (UID: \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:34:49.283125 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.283070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv\" (UID: \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:34:49.283125 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.283116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-plugins-config-volume\") pod \"payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5\" (UID: \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\") " pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:34:49.384301 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.384210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbd9t\" (UniqueName: \"kubernetes.io/projected/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-kube-api-access-jbd9t\") pod \"payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv\" (UID: \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:34:49.384301 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.384267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv\" (UID: \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:34:49.384750 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.384309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-plugins-config-volume\") pod \"payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5\" (UID: \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\") " pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:34:49.384750 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.384350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzt5n\" (UniqueName: \"kubernetes.io/projected/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-kube-api-access-zzt5n\") pod \"payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5\" (UID: \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\") " pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:34:49.385016 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.384996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv\" (UID: \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:34:49.385095 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.385077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-plugins-config-volume\") pod \"payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5\" (UID: \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\") " pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:34:49.393257 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.393236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzt5n\" (UniqueName: \"kubernetes.io/projected/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-kube-api-access-zzt5n\") pod \"payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5\" (UID: \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\") " pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:34:49.393395 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.393278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbd9t\" (UniqueName: \"kubernetes.io/projected/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-kube-api-access-jbd9t\") pod \"payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv\" (UID: \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:34:49.469178 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.469150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:34:49.523403 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.523370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:34:49.669190 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.669140    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"]
Jul 24 17:34:49.671959 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:34:49.671924    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64317c75_ecf4_4e8f_a42a_e1fa06acc44c.slice/crio-bb7912c842dd0e460415ec720816f3f5633fe52899b07c82df65ac1522786f96 WatchSource:0}: Error finding container bb7912c842dd0e460415ec720816f3f5633fe52899b07c82df65ac1522786f96: Status 404 returned error can't find the container with id bb7912c842dd0e460415ec720816f3f5633fe52899b07c82df65ac1522786f96
Jul 24 17:34:49.804425 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.804381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv" event={"ID":"64317c75-ecf4-4e8f-a42a-e1fa06acc44c","Type":"ContainerStarted","Data":"c1b4b6b53c3c327aab45d9c1edaa0285188255ac1f3f78afbca5b306ff60817b"}
Jul 24 17:34:49.804588 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.804452    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:34:49.804588 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.804480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv" event={"ID":"64317c75-ecf4-4e8f-a42a-e1fa06acc44c","Type":"ContainerStarted","Data":"bb7912c842dd0e460415ec720816f3f5633fe52899b07c82df65ac1522786f96"}
Jul 24 17:34:49.819550 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.819521    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"]
Jul 24 17:34:49.822786 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:34:49.822752    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c55c0b3_30aa_4ed4_83b4_2f891fc3bfb6.slice/crio-95eeecd249685efae852b3978ab4277c45fc3a3f089a7949e2314b6c85731fd2 WatchSource:0}: Error finding container 95eeecd249685efae852b3978ab4277c45fc3a3f089a7949e2314b6c85731fd2: Status 404 returned error can't find the container with id 95eeecd249685efae852b3978ab4277c45fc3a3f089a7949e2314b6c85731fd2
Jul 24 17:34:49.822786 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:49.822749    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv" podStartSLOduration=0.822733515 podStartE2EDuration="822.733515ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:34:49.822084098 +0000 UTC m=+1893.653336785" watchObservedRunningTime="2026-07-24 17:34:49.822733515 +0000 UTC m=+1893.653986201"
Jul 24 17:34:50.814317 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:50.814285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5" event={"ID":"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6","Type":"ContainerStarted","Data":"0264b0c6b53b61457dabd17933aac799ab31efbdf00affd41ee76276d0fe6f88"}
Jul 24 17:34:50.814839 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:50.814337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5" event={"ID":"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6","Type":"ContainerStarted","Data":"95eeecd249685efae852b3978ab4277c45fc3a3f089a7949e2314b6c85731fd2"}
Jul 24 17:34:50.814839 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:50.814474    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:34:50.829977 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:34:50.829931    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5" podStartSLOduration=1.829915546 podStartE2EDuration="1.829915546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:34:50.82848356 +0000 UTC m=+1894.659736240" watchObservedRunningTime="2026-07-24 17:34:50.829915546 +0000 UTC m=+1894.661168234"
Jul 24 17:35:00.816827 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:00.816793    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:35:01.821212 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:01.821184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:35:21.934944 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:21.934911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:24.751319 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.751278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:24.781788 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.781744    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"]
Jul 24 17:35:24.781992 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.781966    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5" podUID="9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6" containerName="payload-processing" containerID="cri-o://0264b0c6b53b61457dabd17933aac799ab31efbdf00affd41ee76276d0fe6f88" gracePeriod=30
Jul 24 17:35:24.792762 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.792733    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"]
Jul 24 17:35:24.793532 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.793503    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv" podUID="64317c75-ecf4-4e8f-a42a-e1fa06acc44c" containerName="payload-pre-processing" containerID="cri-o://c1b4b6b53c3c327aab45d9c1edaa0285188255ac1f3f78afbca5b306ff60817b" gracePeriod=30
Jul 24 17:35:24.824388 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.824363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:24.924781 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.924748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:24.976470 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.976438    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6" containerID="0264b0c6b53b61457dabd17933aac799ab31efbdf00affd41ee76276d0fe6f88" exitCode=0
Jul 24 17:35:24.976680 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.976574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5" event={"ID":"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6","Type":"ContainerDied","Data":"0264b0c6b53b61457dabd17933aac799ab31efbdf00affd41ee76276d0fe6f88"}
Jul 24 17:35:24.978372 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.978350    2563 generic.go:356] "Generic (PLEG): container finished" podID="64317c75-ecf4-4e8f-a42a-e1fa06acc44c" containerID="c1b4b6b53c3c327aab45d9c1edaa0285188255ac1f3f78afbca5b306ff60817b" exitCode=0
Jul 24 17:35:24.978503 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:24.978383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv" event={"ID":"64317c75-ecf4-4e8f-a42a-e1fa06acc44c","Type":"ContainerDied","Data":"c1b4b6b53c3c327aab45d9c1edaa0285188255ac1f3f78afbca5b306ff60817b"}
Jul 24 17:35:25.037457 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.037433    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:35:25.065631 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.065609    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:35:25.104599 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.104574    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-plugins-config-volume\") pod \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\" (UID: \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\") "
Jul 24 17:35:25.104765 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.104642    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzt5n\" (UniqueName: \"kubernetes.io/projected/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-kube-api-access-zzt5n\") pod \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\" (UID: \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\") "
Jul 24 17:35:25.104765 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.104715    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-plugins-config-volume\") pod \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\" (UID: \"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6\") "
Jul 24 17:35:25.104860 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.104773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbd9t\" (UniqueName: \"kubernetes.io/projected/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-kube-api-access-jbd9t\") pod \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\" (UID: \"64317c75-ecf4-4e8f-a42a-e1fa06acc44c\") "
Jul 24 17:35:25.104981 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.104955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64317c75-ecf4-4e8f-a42a-e1fa06acc44c" (UID: "64317c75-ecf4-4e8f-a42a-e1fa06acc44c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:35:25.105135 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.105115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6" (UID: "9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:35:25.106917 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.106898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-kube-api-access-zzt5n" (OuterVolumeSpecName: "kube-api-access-zzt5n") pod "9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6" (UID: "9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6"). InnerVolumeSpecName "kube-api-access-zzt5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:35:25.106995 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.106937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-kube-api-access-jbd9t" (OuterVolumeSpecName: "kube-api-access-jbd9t") pod "64317c75-ecf4-4e8f-a42a-e1fa06acc44c" (UID: "64317c75-ecf4-4e8f-a42a-e1fa06acc44c"). InnerVolumeSpecName "kube-api-access-jbd9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:35:25.206109 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.206073    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbd9t\" (UniqueName: \"kubernetes.io/projected/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-kube-api-access-jbd9t\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:35:25.206109 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.206102    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64317c75-ecf4-4e8f-a42a-e1fa06acc44c-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:35:25.206109 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.206115    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzt5n\" (UniqueName: \"kubernetes.io/projected/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-kube-api-access-zzt5n\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:35:25.206328 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.206128    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:35:25.985161 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.985138    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"
Jul 24 17:35:25.985602 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.985195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5" event={"ID":"9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6","Type":"ContainerDied","Data":"95eeecd249685efae852b3978ab4277c45fc3a3f089a7949e2314b6c85731fd2"}
Jul 24 17:35:25.985602 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.985264    2563 scope.go:117] "RemoveContainer" containerID="0264b0c6b53b61457dabd17933aac799ab31efbdf00affd41ee76276d0fe6f88"
Jul 24 17:35:25.986700 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.986681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv" event={"ID":"64317c75-ecf4-4e8f-a42a-e1fa06acc44c","Type":"ContainerDied","Data":"bb7912c842dd0e460415ec720816f3f5633fe52899b07c82df65ac1522786f96"}
Jul 24 17:35:25.986762 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.986705    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"
Jul 24 17:35:25.993859 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:25.993843    2563 scope.go:117] "RemoveContainer" containerID="c1b4b6b53c3c327aab45d9c1edaa0285188255ac1f3f78afbca5b306ff60817b"
Jul 24 17:35:26.012635 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:26.012613    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"]
Jul 24 17:35:26.021112 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:26.021090    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74f5d728-d5565f77d-jwds5"]
Jul 24 17:35:26.033292 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:26.033269    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"]
Jul 24 17:35:26.037621 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:26.037601    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74f5d728-5d67864d96-9dzxv"]
Jul 24 17:35:26.687398 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:26.687362    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64317c75-ecf4-4e8f-a42a-e1fa06acc44c" path="/var/lib/kubelet/pods/64317c75-ecf4-4e8f-a42a-e1fa06acc44c/volumes"
Jul 24 17:35:26.687735 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:26.687721    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6" path="/var/lib/kubelet/pods/9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6/volumes"
Jul 24 17:35:29.151076 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:29.151044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:30.386492 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:30.386440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:30.621912 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:30.621859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:30.837705 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:30.837672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:31.162835 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:31.162747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:31.393466 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:31.393427    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:33.750641 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:33.750609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:33.844379 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:33.844352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:33.944785 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:33.944753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:34.305021 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.304985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:34.473430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.473401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:34.570431 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.570356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:34.599730 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.599701    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"]
Jul 24 17:35:34.600052 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.600041    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6" containerName="payload-processing"
Jul 24 17:35:34.600094 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.600053    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6" containerName="payload-processing"
Jul 24 17:35:34.600094 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.600070    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64317c75-ecf4-4e8f-a42a-e1fa06acc44c" containerName="payload-pre-processing"
Jul 24 17:35:34.600094 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.600075    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="64317c75-ecf4-4e8f-a42a-e1fa06acc44c" containerName="payload-pre-processing"
Jul 24 17:35:34.600199 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.600138    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c55c0b3-30aa-4ed4-83b4-2f891fc3bfb6" containerName="payload-processing"
Jul 24 17:35:34.600199 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.600150    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="64317c75-ecf4-4e8f-a42a-e1fa06acc44c" containerName="payload-pre-processing"
Jul 24 17:35:34.604846 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.604827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:35:34.607172 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.607124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-72534233-dockercfg-h4zr4\""
Jul 24 17:35:34.607336 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.607320    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-72534233\""
Jul 24 17:35:34.612860 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.612839    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"]
Jul 24 17:35:34.658419 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.658391    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"]
Jul 24 17:35:34.661626 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.661612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:34.671970 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.671946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"]
Jul 24 17:35:34.687177 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.687144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcrr\" (UniqueName: \"kubernetes.io/projected/bbd2146d-bdfd-4969-8b17-94a2da748198-kube-api-access-8tcrr\") pod \"payload-processing-e2e-derive-72534233-696b549486-gxrw5\" (UID: \"bbd2146d-bdfd-4969-8b17-94a2da748198\") " pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:35:34.687293 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.687248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2146d-bdfd-4969-8b17-94a2da748198-plugins-config-volume\") pod \"payload-processing-e2e-derive-72534233-696b549486-gxrw5\" (UID: \"bbd2146d-bdfd-4969-8b17-94a2da748198\") " pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:35:34.788300 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.788263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2146d-bdfd-4969-8b17-94a2da748198-plugins-config-volume\") pod \"payload-processing-e2e-derive-72534233-696b549486-gxrw5\" (UID: \"bbd2146d-bdfd-4969-8b17-94a2da748198\") " pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:35:34.788749 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.788350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcrr\" (UniqueName: \"kubernetes.io/projected/bbd2146d-bdfd-4969-8b17-94a2da748198-kube-api-access-8tcrr\") pod \"payload-processing-e2e-derive-72534233-696b549486-gxrw5\" (UID: \"bbd2146d-bdfd-4969-8b17-94a2da748198\") " pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:35:34.788749 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.788400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xswkb\" (UniqueName: \"kubernetes.io/projected/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-kube-api-access-xswkb\") pod \"payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7\" (UID: \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:34.788749 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.788531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7\" (UID: \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:34.788997 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.788979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2146d-bdfd-4969-8b17-94a2da748198-plugins-config-volume\") pod \"payload-processing-e2e-derive-72534233-696b549486-gxrw5\" (UID: \"bbd2146d-bdfd-4969-8b17-94a2da748198\") " pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:35:34.797793 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.797769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcrr\" (UniqueName: \"kubernetes.io/projected/bbd2146d-bdfd-4969-8b17-94a2da748198-kube-api-access-8tcrr\") pod \"payload-processing-e2e-derive-72534233-696b549486-gxrw5\" (UID: \"bbd2146d-bdfd-4969-8b17-94a2da748198\") " pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:35:34.889119 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.889037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswkb\" (UniqueName: \"kubernetes.io/projected/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-kube-api-access-xswkb\") pod \"payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7\" (UID: \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:34.889119 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.889081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7\" (UID: \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:34.889744 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.889729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7\" (UID: \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:34.897145 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.897120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswkb\" (UniqueName: \"kubernetes.io/projected/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-kube-api-access-xswkb\") pod \"payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7\" (UID: \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:34.915806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.915782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:35:34.972567 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:34.972081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:35.066998 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:35.066972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"]
Jul 24 17:35:35.069152 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:35:35.069125    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd2146d_bdfd_4969_8b17_94a2da748198.slice/crio-8fe3dd14066dd41d5dac52d82678cdc8bcdf23254432c1d2c27bc0bee98dc0de WatchSource:0}: Error finding container 8fe3dd14066dd41d5dac52d82678cdc8bcdf23254432c1d2c27bc0bee98dc0de: Status 404 returned error can't find the container with id 8fe3dd14066dd41d5dac52d82678cdc8bcdf23254432c1d2c27bc0bee98dc0de
Jul 24 17:35:35.122599 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:35.122574    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"]
Jul 24 17:35:35.125685 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:35:35.125641    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfc1a6b_6c5e_42b3_92fe_4de863d0ba54.slice/crio-e2aa08ce0fee2e93464d674614698a3b9a9cbff9925de8cc34aad82f1f0492fe WatchSource:0}: Error finding container e2aa08ce0fee2e93464d674614698a3b9a9cbff9925de8cc34aad82f1f0492fe: Status 404 returned error can't find the container with id e2aa08ce0fee2e93464d674614698a3b9a9cbff9925de8cc34aad82f1f0492fe
Jul 24 17:35:36.035879 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:36.035842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7" event={"ID":"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54","Type":"ContainerStarted","Data":"cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9"}
Jul 24 17:35:36.036302 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:36.036003    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:36.036302 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:36.036048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7" event={"ID":"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54","Type":"ContainerStarted","Data":"e2aa08ce0fee2e93464d674614698a3b9a9cbff9925de8cc34aad82f1f0492fe"}
Jul 24 17:35:36.037445 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:36.037416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5" event={"ID":"bbd2146d-bdfd-4969-8b17-94a2da748198","Type":"ContainerStarted","Data":"73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91"}
Jul 24 17:35:36.037569 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:36.037455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5" event={"ID":"bbd2146d-bdfd-4969-8b17-94a2da748198","Type":"ContainerStarted","Data":"8fe3dd14066dd41d5dac52d82678cdc8bcdf23254432c1d2c27bc0bee98dc0de"}
Jul 24 17:35:36.037569 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:36.037473    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:35:36.051448 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:36.051154    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7" podStartSLOduration=2.051126721 podStartE2EDuration="2.051126721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:35:36.050626657 +0000 UTC m=+1939.881879344" watchObservedRunningTime="2026-07-24 17:35:36.051126721 +0000 UTC m=+1939.882379408"
Jul 24 17:35:36.072294 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:36.072254    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5" podStartSLOduration=2.072241004 podStartE2EDuration="2.072241004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:35:36.071532452 +0000 UTC m=+1939.902785139" watchObservedRunningTime="2026-07-24 17:35:36.072241004 +0000 UTC m=+1939.903493689"
Jul 24 17:35:47.043354 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:47.043324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:35:47.043762 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:35:47.043529    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:36:14.192608 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.192533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:14.213772 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.213717    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"]
Jul 24 17:36:14.214027 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.213956    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5" podUID="bbd2146d-bdfd-4969-8b17-94a2da748198" containerName="payload-processing" containerID="cri-o://73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91" gracePeriod=30
Jul 24 17:36:14.226839 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.226811    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"]
Jul 24 17:36:14.227105 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.227055    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7" podUID="8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54" containerName="payload-pre-processing" containerID="cri-o://cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9" gracePeriod=30
Jul 24 17:36:14.260383 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.260356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:14.361205 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.361178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:14.482692 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.482642    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:36:14.493149 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.493127    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:36:14.501871 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.501849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2146d-bdfd-4969-8b17-94a2da748198-plugins-config-volume\") pod \"bbd2146d-bdfd-4969-8b17-94a2da748198\" (UID: \"bbd2146d-bdfd-4969-8b17-94a2da748198\") "
Jul 24 17:36:14.501988 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.501898    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tcrr\" (UniqueName: \"kubernetes.io/projected/bbd2146d-bdfd-4969-8b17-94a2da748198-kube-api-access-8tcrr\") pod \"bbd2146d-bdfd-4969-8b17-94a2da748198\" (UID: \"bbd2146d-bdfd-4969-8b17-94a2da748198\") "
Jul 24 17:36:14.502292 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.502267    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd2146d-bdfd-4969-8b17-94a2da748198-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbd2146d-bdfd-4969-8b17-94a2da748198" (UID: "bbd2146d-bdfd-4969-8b17-94a2da748198"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:36:14.504551 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.504524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd2146d-bdfd-4969-8b17-94a2da748198-kube-api-access-8tcrr" (OuterVolumeSpecName: "kube-api-access-8tcrr") pod "bbd2146d-bdfd-4969-8b17-94a2da748198" (UID: "bbd2146d-bdfd-4969-8b17-94a2da748198"). InnerVolumeSpecName "kube-api-access-8tcrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:36:14.603255 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.603225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xswkb\" (UniqueName: \"kubernetes.io/projected/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-kube-api-access-xswkb\") pod \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\" (UID: \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\") "
Jul 24 17:36:14.603419 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.603283    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-plugins-config-volume\") pod \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\" (UID: \"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54\") "
Jul 24 17:36:14.603464 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.603445    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd2146d-bdfd-4969-8b17-94a2da748198-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:36:14.603464 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.603455    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tcrr\" (UniqueName: \"kubernetes.io/projected/bbd2146d-bdfd-4969-8b17-94a2da748198-kube-api-access-8tcrr\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:36:14.603691 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.603631    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54" (UID: "8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:36:14.605430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.605411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-kube-api-access-xswkb" (OuterVolumeSpecName: "kube-api-access-xswkb") pod "8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54" (UID: "8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54"). InnerVolumeSpecName "kube-api-access-xswkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:36:14.704706 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.704685    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:36:14.704706 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:14.704706    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xswkb\" (UniqueName: \"kubernetes.io/projected/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54-kube-api-access-xswkb\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:36:15.223189 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.223155    2563 generic.go:356] "Generic (PLEG): container finished" podID="8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54" containerID="cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9" exitCode=0
Jul 24 17:36:15.223624 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.223202    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"
Jul 24 17:36:15.223624 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.223248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7" event={"ID":"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54","Type":"ContainerDied","Data":"cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9"}
Jul 24 17:36:15.223624 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.223302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7" event={"ID":"8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54","Type":"ContainerDied","Data":"e2aa08ce0fee2e93464d674614698a3b9a9cbff9925de8cc34aad82f1f0492fe"}
Jul 24 17:36:15.223624 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.223332    2563 scope.go:117] "RemoveContainer" containerID="cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9"
Jul 24 17:36:15.224708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.224581    2563 generic.go:356] "Generic (PLEG): container finished" podID="bbd2146d-bdfd-4969-8b17-94a2da748198" containerID="73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91" exitCode=0
Jul 24 17:36:15.224708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.224620    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"
Jul 24 17:36:15.224708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.224669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5" event={"ID":"bbd2146d-bdfd-4969-8b17-94a2da748198","Type":"ContainerDied","Data":"73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91"}
Jul 24 17:36:15.224708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.224701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5" event={"ID":"bbd2146d-bdfd-4969-8b17-94a2da748198","Type":"ContainerDied","Data":"8fe3dd14066dd41d5dac52d82678cdc8bcdf23254432c1d2c27bc0bee98dc0de"}
Jul 24 17:36:15.236752 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.236734    2563 scope.go:117] "RemoveContainer" containerID="cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9"
Jul 24 17:36:15.237017 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:36:15.236998    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9\": container with ID starting with cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9 not found: ID does not exist" containerID="cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9"
Jul 24 17:36:15.237062 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.237027    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9"} err="failed to get container status \"cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9\": rpc error: code = NotFound desc = could not find container \"cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9\": container with ID starting with cb66417f0db751eade42bf0ec422247de6bdeb14c6d4b15e2efe1593c70d09f9 not found: ID does not exist"
Jul 24 17:36:15.237062 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.237046    2563 scope.go:117] "RemoveContainer" containerID="73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91"
Jul 24 17:36:15.244690 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.244673    2563 scope.go:117] "RemoveContainer" containerID="73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91"
Jul 24 17:36:15.244967 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:36:15.244948    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91\": container with ID starting with 73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91 not found: ID does not exist" containerID="73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91"
Jul 24 17:36:15.245016 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.244973    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91"} err="failed to get container status \"73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91\": rpc error: code = NotFound desc = could not find container \"73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91\": container with ID starting with 73061a4fedba20acdf70d894770255b4a48cb4bd87f527125baf99ba526e5d91 not found: ID does not exist"
Jul 24 17:36:15.248220 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.248196    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"]
Jul 24 17:36:15.249694 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.249673    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72534233-6d848c4688-d5wx7"]
Jul 24 17:36:15.259983 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.259958    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"]
Jul 24 17:36:15.264986 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:15.264965    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72534233-696b549486-gxrw5"]
Jul 24 17:36:16.688328 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:16.688297    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54" path="/var/lib/kubelet/pods/8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54/volumes"
Jul 24 17:36:16.688688 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:16.688634    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbd2146d-bdfd-4969-8b17-94a2da748198" path="/var/lib/kubelet/pods/bbd2146d-bdfd-4969-8b17-94a2da748198/volumes"
Jul 24 17:36:18.829221 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:18.829187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:18.982070 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:18.982025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:19.224508 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:19.224477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:19.742950 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:19.742921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:19.832514 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:19.832487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:19.932148 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:19.932123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:21.428630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:21.428593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:27.467699 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:27.467664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:27.568726 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:27.568696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:27.671189 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:27.671141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:27.767734 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:27.767642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:28.184594 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:28.184520    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:54.169233 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:36:54.169202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:00.621219 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:00.621179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:00.725498 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:00.725464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:00.823790 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:00.823755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:16.775275 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:16.775241    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:22.653226 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:22.653188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:22.721810 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:22.721777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:22.814262 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:22.814230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:33.367700 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:33.367626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:34.190947 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:34.190914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:41.598910 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:41.598804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:41.710807 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:41.710779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:41.812462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:41.812421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:41.910693 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:41.910582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:42.720628 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:42.720594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:42.821426 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:42.821378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:42.958444 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:42.958411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:44.722192 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:44.722160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:50.534932 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:50.534890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:50.634402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:50.634367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:51.220609 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:51.220579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:57.217685 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:57.217636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:57.316285 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:57.316257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:57.460289 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:57.460248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:57.786373 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:57.786330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:58.057040 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:37:58.056947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:00.736727 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:00.736690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:00.834299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:00.834273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:00.988821 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:00.988733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:01.353959 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.353842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:01.505855 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.505816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:01.590865 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.590822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:01.595458 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.595436    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"]
Jul 24 17:38:01.595872 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.595857    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54" containerName="payload-pre-processing"
Jul 24 17:38:01.595872 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.595872    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54" containerName="payload-pre-processing"
Jul 24 17:38:01.595962 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.595887    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd2146d-bdfd-4969-8b17-94a2da748198" containerName="payload-processing"
Jul 24 17:38:01.595962 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.595893    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd2146d-bdfd-4969-8b17-94a2da748198" containerName="payload-processing"
Jul 24 17:38:01.595962 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.595959    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd2146d-bdfd-4969-8b17-94a2da748198" containerName="payload-processing"
Jul 24 17:38:01.596073 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.595967    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cfc1a6b-6c5e-42b3-92fe-4de863d0ba54" containerName="payload-pre-processing"
Jul 24 17:38:01.599047 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.599025    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:01.601243 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.601224    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4ba182d6\""
Jul 24 17:38:01.610256 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.610235    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"]
Jul 24 17:38:01.637582 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.637554    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"]
Jul 24 17:38:01.641231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.641214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:01.648286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.648257    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4ba182d6-dockercfg-v9fls\""
Jul 24 17:38:01.658591 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.658570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"]
Jul 24 17:38:01.743463 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.743434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdae57-2d37-4a74-a705-94b604928fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79\" (UID: \"f8bdae57-2d37-4a74-a705-94b604928fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:01.743832 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.743480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a454a0-1d6b-4d14-8527-90973a36bb45-plugins-config-volume\") pod \"payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d\" (UID: \"b1a454a0-1d6b-4d14-8527-90973a36bb45\") " pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:01.743832 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.743545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gzv\" (UniqueName: \"kubernetes.io/projected/b1a454a0-1d6b-4d14-8527-90973a36bb45-kube-api-access-q2gzv\") pod \"payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d\" (UID: \"b1a454a0-1d6b-4d14-8527-90973a36bb45\") " pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:01.743832 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.743632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jrz\" (UniqueName: \"kubernetes.io/projected/f8bdae57-2d37-4a74-a705-94b604928fe5-kube-api-access-n4jrz\") pod \"payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79\" (UID: \"f8bdae57-2d37-4a74-a705-94b604928fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:01.844619 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.844584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a454a0-1d6b-4d14-8527-90973a36bb45-plugins-config-volume\") pod \"payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d\" (UID: \"b1a454a0-1d6b-4d14-8527-90973a36bb45\") " pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:01.844808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.844630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gzv\" (UniqueName: \"kubernetes.io/projected/b1a454a0-1d6b-4d14-8527-90973a36bb45-kube-api-access-q2gzv\") pod \"payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d\" (UID: \"b1a454a0-1d6b-4d14-8527-90973a36bb45\") " pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:01.844808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.844711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jrz\" (UniqueName: \"kubernetes.io/projected/f8bdae57-2d37-4a74-a705-94b604928fe5-kube-api-access-n4jrz\") pod \"payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79\" (UID: \"f8bdae57-2d37-4a74-a705-94b604928fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:01.844808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.844732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdae57-2d37-4a74-a705-94b604928fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79\" (UID: \"f8bdae57-2d37-4a74-a705-94b604928fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:01.845377 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.845350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a454a0-1d6b-4d14-8527-90973a36bb45-plugins-config-volume\") pod \"payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d\" (UID: \"b1a454a0-1d6b-4d14-8527-90973a36bb45\") " pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:01.845497 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.845400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdae57-2d37-4a74-a705-94b604928fe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79\" (UID: \"f8bdae57-2d37-4a74-a705-94b604928fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:01.853780 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.853754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gzv\" (UniqueName: \"kubernetes.io/projected/b1a454a0-1d6b-4d14-8527-90973a36bb45-kube-api-access-q2gzv\") pod \"payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d\" (UID: \"b1a454a0-1d6b-4d14-8527-90973a36bb45\") " pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:01.853875 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.853791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jrz\" (UniqueName: \"kubernetes.io/projected/f8bdae57-2d37-4a74-a705-94b604928fe5-kube-api-access-n4jrz\") pod \"payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79\" (UID: \"f8bdae57-2d37-4a74-a705-94b604928fe5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:01.909556 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.909439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:01.958852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:01.958815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:02.055576 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.055550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"]
Jul 24 17:38:02.058411 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:38:02.058381    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bdae57_2d37_4a74_a705_94b604928fe5.slice/crio-12da5f619784cc6fb5dce23dd4566e7c0a010f32169f7042d45ec34e91bfa19e WatchSource:0}: Error finding container 12da5f619784cc6fb5dce23dd4566e7c0a010f32169f7042d45ec34e91bfa19e: Status 404 returned error can't find the container with id 12da5f619784cc6fb5dce23dd4566e7c0a010f32169f7042d45ec34e91bfa19e
Jul 24 17:38:02.109324 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.109301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"]
Jul 24 17:38:02.111563 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:38:02.111537    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a454a0_1d6b_4d14_8527_90973a36bb45.slice/crio-e5fcd65ca01527c3e3120a623996539fd3deccf8e683e042a246c586fad801a1 WatchSource:0}: Error finding container e5fcd65ca01527c3e3120a623996539fd3deccf8e683e042a246c586fad801a1: Status 404 returned error can't find the container with id e5fcd65ca01527c3e3120a623996539fd3deccf8e683e042a246c586fad801a1
Jul 24 17:38:02.705903 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.705628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79" event={"ID":"f8bdae57-2d37-4a74-a705-94b604928fe5","Type":"ContainerStarted","Data":"fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418"}
Jul 24 17:38:02.705903 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.705716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:02.705903 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.705741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79" event={"ID":"f8bdae57-2d37-4a74-a705-94b604928fe5","Type":"ContainerStarted","Data":"12da5f619784cc6fb5dce23dd4566e7c0a010f32169f7042d45ec34e91bfa19e"}
Jul 24 17:38:02.707817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.707786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d" event={"ID":"b1a454a0-1d6b-4d14-8527-90973a36bb45","Type":"ContainerStarted","Data":"ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c"}
Jul 24 17:38:02.707961 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.707848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d" event={"ID":"b1a454a0-1d6b-4d14-8527-90973a36bb45","Type":"ContainerStarted","Data":"e5fcd65ca01527c3e3120a623996539fd3deccf8e683e042a246c586fad801a1"}
Jul 24 17:38:02.707961 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.707878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:02.729227 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.729174    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79" podStartSLOduration=1.729154606 podStartE2EDuration="1.729154606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:38:02.727166938 +0000 UTC m=+2086.558419627" watchObservedRunningTime="2026-07-24 17:38:02.729154606 +0000 UTC m=+2086.560407299"
Jul 24 17:38:02.754683 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:02.754621    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d" podStartSLOduration=1.7546074360000001 podStartE2EDuration="1.754607436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:38:02.753641446 +0000 UTC m=+2086.584894134" watchObservedRunningTime="2026-07-24 17:38:02.754607436 +0000 UTC m=+2086.585860122"
Jul 24 17:38:05.241486 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:05.241433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:05.940895 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:05.940865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:06.039663 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:06.039614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:06.408225 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:06.408193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:12.239472 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:12.239424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:12.300875 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:12.300843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:13.714468 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:13.714432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:13.714899 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:13.714517    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:16.729880 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:16.729854    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:38:16.736380 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:16.736359    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:38:21.364106 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:21.364073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:21.463840 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:21.463817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:23.348852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.348820    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:23.386520 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.386490    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"]
Jul 24 17:38:23.386761 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.386734    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d" podUID="b1a454a0-1d6b-4d14-8527-90973a36bb45" containerName="payload-processing" containerID="cri-o://ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c" gracePeriod=30
Jul 24 17:38:23.393363 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.393338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"]
Jul 24 17:38:23.393542 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.393520    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79" podUID="f8bdae57-2d37-4a74-a705-94b604928fe5" containerName="payload-pre-processing" containerID="cri-o://fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418" gracePeriod=30
Jul 24 17:38:23.426239 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.426153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:23.507518 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.507493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:23.637492 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.637468    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:23.665422 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.665399    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:23.732324 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.732299    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a454a0-1d6b-4d14-8527-90973a36bb45-plugins-config-volume\") pod \"b1a454a0-1d6b-4d14-8527-90973a36bb45\" (UID: \"b1a454a0-1d6b-4d14-8527-90973a36bb45\") "
Jul 24 17:38:23.732464 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.732347    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2gzv\" (UniqueName: \"kubernetes.io/projected/b1a454a0-1d6b-4d14-8527-90973a36bb45-kube-api-access-q2gzv\") pod \"b1a454a0-1d6b-4d14-8527-90973a36bb45\" (UID: \"b1a454a0-1d6b-4d14-8527-90973a36bb45\") "
Jul 24 17:38:23.732704 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.732681    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a454a0-1d6b-4d14-8527-90973a36bb45-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1a454a0-1d6b-4d14-8527-90973a36bb45" (UID: "b1a454a0-1d6b-4d14-8527-90973a36bb45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:38:23.734449 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.734430    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a454a0-1d6b-4d14-8527-90973a36bb45-kube-api-access-q2gzv" (OuterVolumeSpecName: "kube-api-access-q2gzv") pod "b1a454a0-1d6b-4d14-8527-90973a36bb45" (UID: "b1a454a0-1d6b-4d14-8527-90973a36bb45"). InnerVolumeSpecName "kube-api-access-q2gzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:38:23.807159 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.807125    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1a454a0-1d6b-4d14-8527-90973a36bb45" containerID="ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c" exitCode=0
Jul 24 17:38:23.807286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.807166    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"
Jul 24 17:38:23.807286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.807168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d" event={"ID":"b1a454a0-1d6b-4d14-8527-90973a36bb45","Type":"ContainerDied","Data":"ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c"}
Jul 24 17:38:23.807286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.807214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d" event={"ID":"b1a454a0-1d6b-4d14-8527-90973a36bb45","Type":"ContainerDied","Data":"e5fcd65ca01527c3e3120a623996539fd3deccf8e683e042a246c586fad801a1"}
Jul 24 17:38:23.807286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.807236    2563 scope.go:117] "RemoveContainer" containerID="ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c"
Jul 24 17:38:23.808643 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.808614    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8bdae57-2d37-4a74-a705-94b604928fe5" containerID="fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418" exitCode=0
Jul 24 17:38:23.808773 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.808691    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"
Jul 24 17:38:23.808773 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.808693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79" event={"ID":"f8bdae57-2d37-4a74-a705-94b604928fe5","Type":"ContainerDied","Data":"fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418"}
Jul 24 17:38:23.808860 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.808797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79" event={"ID":"f8bdae57-2d37-4a74-a705-94b604928fe5","Type":"ContainerDied","Data":"12da5f619784cc6fb5dce23dd4566e7c0a010f32169f7042d45ec34e91bfa19e"}
Jul 24 17:38:23.816400 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.816385    2563 scope.go:117] "RemoveContainer" containerID="ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c"
Jul 24 17:38:23.816702 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:38:23.816684    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c\": container with ID starting with ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c not found: ID does not exist" containerID="ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c"
Jul 24 17:38:23.816770 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.816711    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c"} err="failed to get container status \"ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c\": rpc error: code = NotFound desc = could not find container \"ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c\": container with ID starting with ec9c1aafd1e8abce38d9148ec8536887836e641655266c4b618bfbbc3f5ccf3c not found: ID does not exist"
Jul 24 17:38:23.816770 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.816729    2563 scope.go:117] "RemoveContainer" containerID="fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418"
Jul 24 17:38:23.824461 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.824444    2563 scope.go:117] "RemoveContainer" containerID="fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418"
Jul 24 17:38:23.824723 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:38:23.824702    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418\": container with ID starting with fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418 not found: ID does not exist" containerID="fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418"
Jul 24 17:38:23.824813 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.824730    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418"} err="failed to get container status \"fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418\": rpc error: code = NotFound desc = could not find container \"fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418\": container with ID starting with fb287d9563e3a67fe608dfea52a1c9fb4f106e0c42d0b1a94328faee6200c418 not found: ID does not exist"
Jul 24 17:38:23.828036 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.828014    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"]
Jul 24 17:38:23.832025 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.832007    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4ba182d6-55dfccbb66-v784d"]
Jul 24 17:38:23.833521 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.833503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4jrz\" (UniqueName: \"kubernetes.io/projected/f8bdae57-2d37-4a74-a705-94b604928fe5-kube-api-access-n4jrz\") pod \"f8bdae57-2d37-4a74-a705-94b604928fe5\" (UID: \"f8bdae57-2d37-4a74-a705-94b604928fe5\") "
Jul 24 17:38:23.833585 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.833545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdae57-2d37-4a74-a705-94b604928fe5-plugins-config-volume\") pod \"f8bdae57-2d37-4a74-a705-94b604928fe5\" (UID: \"f8bdae57-2d37-4a74-a705-94b604928fe5\") "
Jul 24 17:38:23.833712 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.833699    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a454a0-1d6b-4d14-8527-90973a36bb45-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:38:23.833778 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.833713    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2gzv\" (UniqueName: \"kubernetes.io/projected/b1a454a0-1d6b-4d14-8527-90973a36bb45-kube-api-access-q2gzv\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:38:23.833910 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.833890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bdae57-2d37-4a74-a705-94b604928fe5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8bdae57-2d37-4a74-a705-94b604928fe5" (UID: "f8bdae57-2d37-4a74-a705-94b604928fe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:38:23.835569 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.835550    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bdae57-2d37-4a74-a705-94b604928fe5-kube-api-access-n4jrz" (OuterVolumeSpecName: "kube-api-access-n4jrz") pod "f8bdae57-2d37-4a74-a705-94b604928fe5" (UID: "f8bdae57-2d37-4a74-a705-94b604928fe5"). InnerVolumeSpecName "kube-api-access-n4jrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:38:23.934395 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.934342    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jrz\" (UniqueName: \"kubernetes.io/projected/f8bdae57-2d37-4a74-a705-94b604928fe5-kube-api-access-n4jrz\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:38:23.934395 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:23.934363    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdae57-2d37-4a74-a705-94b604928fe5-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:38:24.129073 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:24.129048    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"]
Jul 24 17:38:24.132508 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:24.132486    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ba182d6-789699b656-6qb79"]
Jul 24 17:38:24.687882 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:24.687815    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a454a0-1d6b-4d14-8527-90973a36bb45" path="/var/lib/kubelet/pods/b1a454a0-1d6b-4d14-8527-90973a36bb45/volumes"
Jul 24 17:38:24.688207 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:24.688137    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8bdae57-2d37-4a74-a705-94b604928fe5" path="/var/lib/kubelet/pods/f8bdae57-2d37-4a74-a705-94b604928fe5/volumes"
Jul 24 17:38:27.722475 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:27.722439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:28.114551 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:28.114468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:34.409552 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:34.409513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:34.514192 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:34.514159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:34.711270 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:34.711194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:34.979137 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:34.979106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:35.117212 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:35.117174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:35.229482 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:35.229396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:35.328865 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:35.328834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:35.910745 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:35.910698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:36.110071 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:36.110018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:41.007588 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:41.007554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:41.113850 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:41.113823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:41.210893 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:41.210861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:41.379302 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:41.379225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:45.117039 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:45.117008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:47.087889 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:47.087861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:47.187837 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:47.187805    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:53.956148 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:53.956116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:54.247154 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:54.247092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:54.400970 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:54.400934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:57.055244 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.055209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:57.149218 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.149187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:57.248736 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.248701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:57.832085 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.832053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:57.932990 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.932960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:57.933330 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.933310    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"]
Jul 24 17:38:57.933680 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.933668    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a454a0-1d6b-4d14-8527-90973a36bb45" containerName="payload-processing"
Jul 24 17:38:57.933721 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.933682    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a454a0-1d6b-4d14-8527-90973a36bb45" containerName="payload-processing"
Jul 24 17:38:57.933721 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.933692    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8bdae57-2d37-4a74-a705-94b604928fe5" containerName="payload-pre-processing"
Jul 24 17:38:57.933721 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.933697    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8bdae57-2d37-4a74-a705-94b604928fe5" containerName="payload-pre-processing"
Jul 24 17:38:57.933816 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.933760    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a454a0-1d6b-4d14-8527-90973a36bb45" containerName="payload-processing"
Jul 24 17:38:57.933816 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.933769    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8bdae57-2d37-4a74-a705-94b604928fe5" containerName="payload-pre-processing"
Jul 24 17:38:57.936785 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.936771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:38:57.939349 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.939327    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-47dfc7db\""
Jul 24 17:38:57.949491 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.949471    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"]
Jul 24 17:38:57.971956 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.971920    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"]
Jul 24 17:38:57.975257 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.975236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:38:57.977526 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.977506    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-47dfc7db-dockercfg-vsk97\""
Jul 24 17:38:57.986961 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:57.986940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"]
Jul 24 17:38:58.015268 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.015236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:58.030511 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.030485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdbw\" (UniqueName: \"kubernetes.io/projected/42556052-4023-4d66-9400-9634e868bafe-kube-api-access-kjdbw\") pod \"payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj\" (UID: \"42556052-4023-4d66-9400-9634e868bafe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:38:58.030619 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.030519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42556052-4023-4d66-9400-9634e868bafe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj\" (UID: \"42556052-4023-4d66-9400-9634e868bafe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:38:58.131602 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.131526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdbw\" (UniqueName: \"kubernetes.io/projected/42556052-4023-4d66-9400-9634e868bafe-kube-api-access-kjdbw\") pod \"payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj\" (UID: \"42556052-4023-4d66-9400-9634e868bafe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:38:58.131602 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.131562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrdr\" (UniqueName: \"kubernetes.io/projected/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-kube-api-access-kdrdr\") pod \"payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8\" (UID: \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\") " pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:38:58.131602 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.131584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42556052-4023-4d66-9400-9634e868bafe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj\" (UID: \"42556052-4023-4d66-9400-9634e868bafe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:38:58.132025 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.131705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-plugins-config-volume\") pod \"payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8\" (UID: \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\") " pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:38:58.132239 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.132222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42556052-4023-4d66-9400-9634e868bafe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj\" (UID: \"42556052-4023-4d66-9400-9634e868bafe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:38:58.140548 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.140527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdbw\" (UniqueName: \"kubernetes.io/projected/42556052-4023-4d66-9400-9634e868bafe-kube-api-access-kjdbw\") pod \"payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj\" (UID: \"42556052-4023-4d66-9400-9634e868bafe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:38:58.232930 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.232900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrdr\" (UniqueName: \"kubernetes.io/projected/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-kube-api-access-kdrdr\") pod \"payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8\" (UID: \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\") " pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:38:58.232930 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.232937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-plugins-config-volume\") pod \"payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8\" (UID: \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\") " pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:38:58.233602 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.233585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-plugins-config-volume\") pod \"payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8\" (UID: \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\") " pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:38:58.241099 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.241075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrdr\" (UniqueName: \"kubernetes.io/projected/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-kube-api-access-kdrdr\") pod \"payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8\" (UID: \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\") " pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:38:58.247605 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.247539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:38:58.285636 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.285589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:38:58.405876 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.405846    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"]
Jul 24 17:38:58.409340 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:38:58.409311    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42556052_4023_4d66_9400_9634e868bafe.slice/crio-fec2cb8ea632f3252f4c7b7b77b40265916b945298707981272e1aecab3c9751 WatchSource:0}: Error finding container fec2cb8ea632f3252f4c7b7b77b40265916b945298707981272e1aecab3c9751: Status 404 returned error can't find the container with id fec2cb8ea632f3252f4c7b7b77b40265916b945298707981272e1aecab3c9751
Jul 24 17:38:58.436605 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.436580    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"]
Jul 24 17:38:58.442780 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:38:58.442754    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaa1df2_d6eb_4835_82f4_b3e52a19e484.slice/crio-6fafddb4cbaea570e1fb9c4900e62afee03bafea83a920beab885589c5efc8c5 WatchSource:0}: Error finding container 6fafddb4cbaea570e1fb9c4900e62afee03bafea83a920beab885589c5efc8c5: Status 404 returned error can't find the container with id 6fafddb4cbaea570e1fb9c4900e62afee03bafea83a920beab885589c5efc8c5
Jul 24 17:38:58.978402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.978361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8" event={"ID":"1aaa1df2-d6eb-4835-82f4-b3e52a19e484","Type":"ContainerStarted","Data":"902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1"}
Jul 24 17:38:58.978609 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.978423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8" event={"ID":"1aaa1df2-d6eb-4835-82f4-b3e52a19e484","Type":"ContainerStarted","Data":"6fafddb4cbaea570e1fb9c4900e62afee03bafea83a920beab885589c5efc8c5"}
Jul 24 17:38:58.978609 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.978476    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:38:58.980015 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.979986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj" event={"ID":"42556052-4023-4d66-9400-9634e868bafe","Type":"ContainerStarted","Data":"dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff"}
Jul 24 17:38:58.980120 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.980030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj" event={"ID":"42556052-4023-4d66-9400-9634e868bafe","Type":"ContainerStarted","Data":"fec2cb8ea632f3252f4c7b7b77b40265916b945298707981272e1aecab3c9751"}
Jul 24 17:38:58.980120 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:58.980052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:38:59.003049 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:59.002999    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8" podStartSLOduration=2.002982652 podStartE2EDuration="2.002982652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:38:59.000160646 +0000 UTC m=+2142.831413359" watchObservedRunningTime="2026-07-24 17:38:59.002982652 +0000 UTC m=+2142.834235340"
Jul 24 17:38:59.028953 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:38:59.028907    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj" podStartSLOduration=2.028893113 podStartE2EDuration="2.028893113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:38:59.027616868 +0000 UTC m=+2142.858869554" watchObservedRunningTime="2026-07-24 17:38:59.028893113 +0000 UTC m=+2142.860145808"
Jul 24 17:39:01.575480 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:01.575448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:39:09.987565 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:09.987528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:39:09.988002 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:09.987607    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:39:22.863444 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.863410    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"]
Jul 24 17:39:22.866830 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.866815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:22.870995 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.870968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-47dfc7db\"/\"default-dockercfg-8gg9l\""
Jul 24 17:39:22.870995 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.870968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-47dfc7db\"/\"e2e-lifecycle-model-47dfc7db-kserve-self-signed-certs\""
Jul 24 17:39:22.871188 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.870970    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-47dfc7db\"/\"kube-root-ca.crt\""
Jul 24 17:39:22.871188 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.870997    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-47dfc7db\"/\"openshift-service-ca.crt\""
Jul 24 17:39:22.877937 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.877914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"]
Jul 24 17:39:22.951676 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.951617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-dshm\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:22.951865 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.951682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxrrr\" (UniqueName: \"kubernetes.io/projected/781e551d-edf3-49e6-8dc4-2f95b53bee51-kube-api-access-nxrrr\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:22.951865 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.951796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-home\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:22.951865 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.951859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/781e551d-edf3-49e6-8dc4-2f95b53bee51-tls-certs\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:22.952027 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.951989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-model-cache\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:22.952067 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.952048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-tmp-dir\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:22.954806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:22.954777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:39:23.052916 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.052887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-dshm\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.053077 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.052921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxrrr\" (UniqueName: \"kubernetes.io/projected/781e551d-edf3-49e6-8dc4-2f95b53bee51-kube-api-access-nxrrr\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.053077 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.052962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-home\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.053077 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.053004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/781e551d-edf3-49e6-8dc4-2f95b53bee51-tls-certs\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.053077 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.053053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-model-cache\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.053295 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.053091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-tmp-dir\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.053434 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.053393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-home\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.053547 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.053532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-tmp-dir\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.053547 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.053535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-model-cache\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.055479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.055457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-dshm\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.055709 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.055689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/781e551d-edf3-49e6-8dc4-2f95b53bee51-tls-certs\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.076380 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.076359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxrrr\" (UniqueName: \"kubernetes.io/projected/781e551d-edf3-49e6-8dc4-2f95b53bee51-kube-api-access-nxrrr\") pod \"e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") " pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.176481 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.176403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:23.309668 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:23.309611    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"]
Jul 24 17:39:23.313104 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:39:23.313080    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod781e551d_edf3_49e6_8dc4_2f95b53bee51.slice/crio-0b1b42c304fcdc2b86d838b093fe5245de78ad3d7c1f56e3d49f6cee107eea99 WatchSource:0}: Error finding container 0b1b42c304fcdc2b86d838b093fe5245de78ad3d7c1f56e3d49f6cee107eea99: Status 404 returned error can't find the container with id 0b1b42c304fcdc2b86d838b093fe5245de78ad3d7c1f56e3d49f6cee107eea99
Jul 24 17:39:24.094315 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:24.094283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc" event={"ID":"781e551d-edf3-49e6-8dc4-2f95b53bee51","Type":"ContainerStarted","Data":"e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b"}
Jul 24 17:39:24.094817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:24.094329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc" event={"ID":"781e551d-edf3-49e6-8dc4-2f95b53bee51","Type":"ContainerStarted","Data":"0b1b42c304fcdc2b86d838b093fe5245de78ad3d7c1f56e3d49f6cee107eea99"}
Jul 24 17:39:24.110924 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:24.110839    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc" podStartSLOduration=2.110818756 podStartE2EDuration="2.110818756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:39:24.11033545 +0000 UTC m=+2167.941588139" watchObservedRunningTime="2026-07-24 17:39:24.110818756 +0000 UTC m=+2167.942071445"
Jul 24 17:39:33.177329 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:33.177285    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:33.177962 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:33.177349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:33.189801 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:33.189779    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:39:34.150357 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:39:34.150327    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:40:32.646030 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:32.645997    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:32.689215 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:32.689190    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"]
Jul 24 17:40:32.689393 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:32.689371    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8" podUID="1aaa1df2-d6eb-4835-82f4-b3e52a19e484" containerName="payload-processing" containerID="cri-o://902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1" gracePeriod=30
Jul 24 17:40:32.694551 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:32.694525    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"]
Jul 24 17:40:32.694729 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:32.694693    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj" podUID="42556052-4023-4d66-9400-9634e868bafe" containerName="payload-pre-processing" containerID="cri-o://dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff" gracePeriod=30
Jul 24 17:40:32.747005 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:32.746982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:32.826358 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:32.826321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:32.964417 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:32.964397    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:40:32.967284 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:32.967265    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:40:33.081874 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.081844    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42556052-4023-4d66-9400-9634e868bafe-plugins-config-volume\") pod \"42556052-4023-4d66-9400-9634e868bafe\" (UID: \"42556052-4023-4d66-9400-9634e868bafe\") "
Jul 24 17:40:33.082043 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.081906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdrdr\" (UniqueName: \"kubernetes.io/projected/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-kube-api-access-kdrdr\") pod \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\" (UID: \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\") "
Jul 24 17:40:33.082043 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.081940    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjdbw\" (UniqueName: \"kubernetes.io/projected/42556052-4023-4d66-9400-9634e868bafe-kube-api-access-kjdbw\") pod \"42556052-4023-4d66-9400-9634e868bafe\" (UID: \"42556052-4023-4d66-9400-9634e868bafe\") "
Jul 24 17:40:33.082043 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.081974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-plugins-config-volume\") pod \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\" (UID: \"1aaa1df2-d6eb-4835-82f4-b3e52a19e484\") "
Jul 24 17:40:33.082349 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.082319    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42556052-4023-4d66-9400-9634e868bafe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42556052-4023-4d66-9400-9634e868bafe" (UID: "42556052-4023-4d66-9400-9634e868bafe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:40:33.082440 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.082392    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1aaa1df2-d6eb-4835-82f4-b3e52a19e484" (UID: "1aaa1df2-d6eb-4835-82f4-b3e52a19e484"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:40:33.084148 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.084122    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-kube-api-access-kdrdr" (OuterVolumeSpecName: "kube-api-access-kdrdr") pod "1aaa1df2-d6eb-4835-82f4-b3e52a19e484" (UID: "1aaa1df2-d6eb-4835-82f4-b3e52a19e484"). InnerVolumeSpecName "kube-api-access-kdrdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:40:33.084288 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.084269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42556052-4023-4d66-9400-9634e868bafe-kube-api-access-kjdbw" (OuterVolumeSpecName: "kube-api-access-kjdbw") pod "42556052-4023-4d66-9400-9634e868bafe" (UID: "42556052-4023-4d66-9400-9634e868bafe"). InnerVolumeSpecName "kube-api-access-kjdbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:40:33.183446 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.183415    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjdbw\" (UniqueName: \"kubernetes.io/projected/42556052-4023-4d66-9400-9634e868bafe-kube-api-access-kjdbw\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:40:33.183446 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.183445    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:40:33.183446 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.183456    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42556052-4023-4d66-9400-9634e868bafe-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:40:33.183699 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.183464    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdrdr\" (UniqueName: \"kubernetes.io/projected/1aaa1df2-d6eb-4835-82f4-b3e52a19e484-kube-api-access-kdrdr\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:40:33.410555 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.410519    2563 generic.go:356] "Generic (PLEG): container finished" podID="42556052-4023-4d66-9400-9634e868bafe" containerID="dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff" exitCode=0
Jul 24 17:40:33.410739 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.410574    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"
Jul 24 17:40:33.410739 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.410614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj" event={"ID":"42556052-4023-4d66-9400-9634e868bafe","Type":"ContainerDied","Data":"dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff"}
Jul 24 17:40:33.410739 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.410717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj" event={"ID":"42556052-4023-4d66-9400-9634e868bafe","Type":"ContainerDied","Data":"fec2cb8ea632f3252f4c7b7b77b40265916b945298707981272e1aecab3c9751"}
Jul 24 17:40:33.410884 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.410739    2563 scope.go:117] "RemoveContainer" containerID="dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff"
Jul 24 17:40:33.412083 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.412059    2563 generic.go:356] "Generic (PLEG): container finished" podID="1aaa1df2-d6eb-4835-82f4-b3e52a19e484" containerID="902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1" exitCode=0
Jul 24 17:40:33.412201 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.412089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8" event={"ID":"1aaa1df2-d6eb-4835-82f4-b3e52a19e484","Type":"ContainerDied","Data":"902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1"}
Jul 24 17:40:33.412201 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.412119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8" event={"ID":"1aaa1df2-d6eb-4835-82f4-b3e52a19e484","Type":"ContainerDied","Data":"6fafddb4cbaea570e1fb9c4900e62afee03bafea83a920beab885589c5efc8c5"}
Jul 24 17:40:33.412336 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.412130    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"
Jul 24 17:40:33.420577 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.420562    2563 scope.go:117] "RemoveContainer" containerID="dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff"
Jul 24 17:40:33.420851 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:40:33.420832    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff\": container with ID starting with dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff not found: ID does not exist" containerID="dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff"
Jul 24 17:40:33.420922 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.420858    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff"} err="failed to get container status \"dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff\": rpc error: code = NotFound desc = could not find container \"dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff\": container with ID starting with dc5cbf27032736c1a5fa68760cc51ab9407e8e2532fe5512ab1b32a9a81945ff not found: ID does not exist"
Jul 24 17:40:33.420922 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.420874    2563 scope.go:117] "RemoveContainer" containerID="902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1"
Jul 24 17:40:33.428872 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.428853    2563 scope.go:117] "RemoveContainer" containerID="902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1"
Jul 24 17:40:33.429131 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:40:33.429111    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1\": container with ID starting with 902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1 not found: ID does not exist" containerID="902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1"
Jul 24 17:40:33.429192 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.429140    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1"} err="failed to get container status \"902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1\": rpc error: code = NotFound desc = could not find container \"902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1\": container with ID starting with 902aad699a9fdb752232b41a8b6aead7cd03e9f9cb361d1346814424feed53b1 not found: ID does not exist"
Jul 24 17:40:33.463548 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.463527    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"]
Jul 24 17:40:33.485057 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.485028    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47dfc7db-7d7b97cbbd-8lwrj"]
Jul 24 17:40:33.509255 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.509229    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"]
Jul 24 17:40:33.514750 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:33.514730    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47dfc7db-7c9994f4db-pp7k8"]
Jul 24 17:40:34.687836 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:34.687801    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aaa1df2-d6eb-4835-82f4-b3e52a19e484" path="/var/lib/kubelet/pods/1aaa1df2-d6eb-4835-82f4-b3e52a19e484/volumes"
Jul 24 17:40:34.688233 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:34.688156    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42556052-4023-4d66-9400-9634e868bafe" path="/var/lib/kubelet/pods/42556052-4023-4d66-9400-9634e868bafe/volumes"
Jul 24 17:40:37.219776 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:37.219696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:40.538240 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:40.538207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:40:45.519754 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:45.519718    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"]
Jul 24 17:40:45.565442 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:40:45.565410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:00.539041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.538982    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc" podUID="781e551d-edf3-49e6-8dc4-2f95b53bee51" containerName="main" containerID="cri-o://e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b" gracePeriod=45
Jul 24 17:41:00.782717 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.782687    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:41:00.926791 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.926712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-model-cache\") pod \"781e551d-edf3-49e6-8dc4-2f95b53bee51\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") "
Jul 24 17:41:00.926791 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.926749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-home\") pod \"781e551d-edf3-49e6-8dc4-2f95b53bee51\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") "
Jul 24 17:41:00.926791 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.926786    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-dshm\") pod \"781e551d-edf3-49e6-8dc4-2f95b53bee51\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") "
Jul 24 17:41:00.927074 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.926814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-tmp-dir\") pod \"781e551d-edf3-49e6-8dc4-2f95b53bee51\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") "
Jul 24 17:41:00.927074 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.926875    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/781e551d-edf3-49e6-8dc4-2f95b53bee51-tls-certs\") pod \"781e551d-edf3-49e6-8dc4-2f95b53bee51\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") "
Jul 24 17:41:00.927074 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.926910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxrrr\" (UniqueName: \"kubernetes.io/projected/781e551d-edf3-49e6-8dc4-2f95b53bee51-kube-api-access-nxrrr\") pod \"781e551d-edf3-49e6-8dc4-2f95b53bee51\" (UID: \"781e551d-edf3-49e6-8dc4-2f95b53bee51\") "
Jul 24 17:41:00.927074 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.927049    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-model-cache" (OuterVolumeSpecName: "model-cache") pod "781e551d-edf3-49e6-8dc4-2f95b53bee51" (UID: "781e551d-edf3-49e6-8dc4-2f95b53bee51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:41:00.927074 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.927031    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-home" (OuterVolumeSpecName: "home") pod "781e551d-edf3-49e6-8dc4-2f95b53bee51" (UID: "781e551d-edf3-49e6-8dc4-2f95b53bee51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:41:00.927299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.927101    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "781e551d-edf3-49e6-8dc4-2f95b53bee51" (UID: "781e551d-edf3-49e6-8dc4-2f95b53bee51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:41:00.927299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.927179    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-model-cache\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:41:00.927299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.927191    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-home\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:41:00.927299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.927199    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-tmp-dir\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:41:00.929202 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.929174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-dshm" (OuterVolumeSpecName: "dshm") pod "781e551d-edf3-49e6-8dc4-2f95b53bee51" (UID: "781e551d-edf3-49e6-8dc4-2f95b53bee51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:41:00.929294 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.929236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781e551d-edf3-49e6-8dc4-2f95b53bee51-kube-api-access-nxrrr" (OuterVolumeSpecName: "kube-api-access-nxrrr") pod "781e551d-edf3-49e6-8dc4-2f95b53bee51" (UID: "781e551d-edf3-49e6-8dc4-2f95b53bee51"). InnerVolumeSpecName "kube-api-access-nxrrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:41:00.929294 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:00.929248    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781e551d-edf3-49e6-8dc4-2f95b53bee51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "781e551d-edf3-49e6-8dc4-2f95b53bee51" (UID: "781e551d-edf3-49e6-8dc4-2f95b53bee51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:41:01.028684 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.028637    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/781e551d-edf3-49e6-8dc4-2f95b53bee51-dshm\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:41:01.028684 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.028679    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/781e551d-edf3-49e6-8dc4-2f95b53bee51-tls-certs\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:41:01.028684 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.028691    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxrrr\" (UniqueName: \"kubernetes.io/projected/781e551d-edf3-49e6-8dc4-2f95b53bee51-kube-api-access-nxrrr\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:41:01.552930 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.552890    2563 generic.go:356] "Generic (PLEG): container finished" podID="781e551d-edf3-49e6-8dc4-2f95b53bee51" containerID="e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b" exitCode=0
Jul 24 17:41:01.553402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.552932    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"
Jul 24 17:41:01.553402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.552979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc" event={"ID":"781e551d-edf3-49e6-8dc4-2f95b53bee51","Type":"ContainerDied","Data":"e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b"}
Jul 24 17:41:01.553402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.553020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc" event={"ID":"781e551d-edf3-49e6-8dc4-2f95b53bee51","Type":"ContainerDied","Data":"0b1b42c304fcdc2b86d838b093fe5245de78ad3d7c1f56e3d49f6cee107eea99"}
Jul 24 17:41:01.553402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.553038    2563 scope.go:117] "RemoveContainer" containerID="e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b"
Jul 24 17:41:01.562191 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.562174    2563 scope.go:117] "RemoveContainer" containerID="e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b"
Jul 24 17:41:01.562475 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:41:01.562456    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b\": container with ID starting with e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b not found: ID does not exist" containerID="e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b"
Jul 24 17:41:01.562532 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.562484    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b"} err="failed to get container status \"e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b\": rpc error: code = NotFound desc = could not find container \"e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b\": container with ID starting with e9067b8ccce0b9d2c908fe0e2cc055688ceb4006d1bce978e75f016839e4ae5b not found: ID does not exist"
Jul 24 17:41:01.573164 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.573137    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"]
Jul 24 17:41:01.579034 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:01.579013    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-47dfc7db/e2e-lifecycle-model-47dfc7db-kserve-566d8cbdff-fnptc"]
Jul 24 17:41:02.687876 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:02.687841    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="781e551d-edf3-49e6-8dc4-2f95b53bee51" path="/var/lib/kubelet/pods/781e551d-edf3-49e6-8dc4-2f95b53bee51/volumes"
Jul 24 17:41:22.217588 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:22.217545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:22.683064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:22.682977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:28.160209 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:28.160179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:42.250870 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:42.250834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:42.316071 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:42.316033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:42.552861 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:42.552759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:42.778165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:42.778137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:43.890128 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:43.890103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:43.991640 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:43.991607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:44.094060 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:44.094032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:44.919737 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:44.919704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:45.060504 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:45.060476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:46.523951 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:46.523921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:52.501739 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:52.501702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:52.602484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:52.602452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:53.189822 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:53.189785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:41:59.665677 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:41:59.665635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:01.106971 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:01.106936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:10.418503 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:10.418467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:10.829421 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:10.829387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:31.790128 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:31.790092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:38.252240 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:38.252209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:38.365280 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:38.365249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:38.641679 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:38.641583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:38.961569 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:38.961493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:39.439604 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:39.439575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:42.210012 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:42.209970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:42.303094 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:42.303061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:42.531824 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:42.531794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:42.636273 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:42.636242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:42.970981 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:42.970938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:43.069472 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.069433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:43.084239 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084209    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"]
Jul 24 17:42:43.084564 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084552    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aaa1df2-d6eb-4835-82f4-b3e52a19e484" containerName="payload-processing"
Jul 24 17:42:43.084623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084565    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aaa1df2-d6eb-4835-82f4-b3e52a19e484" containerName="payload-processing"
Jul 24 17:42:43.084623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084577    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781e551d-edf3-49e6-8dc4-2f95b53bee51" containerName="main"
Jul 24 17:42:43.084623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084584    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="781e551d-edf3-49e6-8dc4-2f95b53bee51" containerName="main"
Jul 24 17:42:43.084623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084596    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42556052-4023-4d66-9400-9634e868bafe" containerName="payload-pre-processing"
Jul 24 17:42:43.084623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084602    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42556052-4023-4d66-9400-9634e868bafe" containerName="payload-pre-processing"
Jul 24 17:42:43.084828 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084664    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aaa1df2-d6eb-4835-82f4-b3e52a19e484" containerName="payload-processing"
Jul 24 17:42:43.084828 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084676    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42556052-4023-4d66-9400-9634e868bafe" containerName="payload-pre-processing"
Jul 24 17:42:43.084828 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.084683    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="781e551d-edf3-49e6-8dc4-2f95b53bee51" containerName="main"
Jul 24 17:42:43.087541 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.087527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:42:43.089681 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.089663    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5e8b71\""
Jul 24 17:42:43.089881 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.089869    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5e8b71-dockercfg-9d2wp\""
Jul 24 17:42:43.101141 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.101119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"]
Jul 24 17:42:43.103748 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.103727    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"]
Jul 24 17:42:43.107172 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.107157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:42:43.118013 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.117989    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"]
Jul 24 17:42:43.193398 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.193353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx\" (UID: \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:42:43.193398 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.193404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ftf6\" (UniqueName: \"kubernetes.io/projected/f5150981-d8d5-4039-99c1-87df1a44df92-kube-api-access-9ftf6\") pod \"payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq\" (UID: \"f5150981-d8d5-4039-99c1-87df1a44df92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:42:43.193642 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.193458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpgfr\" (UniqueName: \"kubernetes.io/projected/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-kube-api-access-vpgfr\") pod \"payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx\" (UID: \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:42:43.193642 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.193537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5150981-d8d5-4039-99c1-87df1a44df92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq\" (UID: \"f5150981-d8d5-4039-99c1-87df1a44df92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:42:43.294673 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.294546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5150981-d8d5-4039-99c1-87df1a44df92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq\" (UID: \"f5150981-d8d5-4039-99c1-87df1a44df92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:42:43.295108 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.294676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx\" (UID: \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:42:43.295108 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.294708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ftf6\" (UniqueName: \"kubernetes.io/projected/f5150981-d8d5-4039-99c1-87df1a44df92-kube-api-access-9ftf6\") pod \"payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq\" (UID: \"f5150981-d8d5-4039-99c1-87df1a44df92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:42:43.295108 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.294773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpgfr\" (UniqueName: \"kubernetes.io/projected/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-kube-api-access-vpgfr\") pod \"payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx\" (UID: \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:42:43.295406 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.295387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5150981-d8d5-4039-99c1-87df1a44df92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq\" (UID: \"f5150981-d8d5-4039-99c1-87df1a44df92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:42:43.295447 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.295416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx\" (UID: \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:42:43.303022 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.302991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpgfr\" (UniqueName: \"kubernetes.io/projected/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-kube-api-access-vpgfr\") pod \"payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx\" (UID: \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:42:43.303141 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.303046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ftf6\" (UniqueName: \"kubernetes.io/projected/f5150981-d8d5-4039-99c1-87df1a44df92-kube-api-access-9ftf6\") pod \"payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq\" (UID: \"f5150981-d8d5-4039-99c1-87df1a44df92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:42:43.398308 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.398269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:42:43.417440 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.417415    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:42:43.548617 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.548538    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"]
Jul 24 17:42:43.552095 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:42:43.552063    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5150981_d8d5_4039_99c1_87df1a44df92.slice/crio-e20670e4135f223126dc69dcfd4f6a07105d4c3dbc800d442890d58c5f290a72 WatchSource:0}: Error finding container e20670e4135f223126dc69dcfd4f6a07105d4c3dbc800d442890d58c5f290a72: Status 404 returned error can't find the container with id e20670e4135f223126dc69dcfd4f6a07105d4c3dbc800d442890d58c5f290a72
Jul 24 17:42:43.580085 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:43.580062    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"]
Jul 24 17:42:43.585836 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:42:43.585805    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ba02b8_f3b0_4f45_95ee_2707a19ef3a1.slice/crio-9d6c60de9ec9ca26571cc3ebe1033bbfb44dab42f5a37174f4c9b6a04bd72d6d WatchSource:0}: Error finding container 9d6c60de9ec9ca26571cc3ebe1033bbfb44dab42f5a37174f4c9b6a04bd72d6d: Status 404 returned error can't find the container with id 9d6c60de9ec9ca26571cc3ebe1033bbfb44dab42f5a37174f4c9b6a04bd72d6d
Jul 24 17:42:44.008743 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:44.008703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx" event={"ID":"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1","Type":"ContainerStarted","Data":"629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd"}
Jul 24 17:42:44.008930 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:44.008771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx" event={"ID":"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1","Type":"ContainerStarted","Data":"9d6c60de9ec9ca26571cc3ebe1033bbfb44dab42f5a37174f4c9b6a04bd72d6d"}
Jul 24 17:42:44.008930 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:44.008815    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:42:44.010532 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:44.010507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq" event={"ID":"f5150981-d8d5-4039-99c1-87df1a44df92","Type":"ContainerStarted","Data":"bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b"}
Jul 24 17:42:44.010698 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:44.010545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq" event={"ID":"f5150981-d8d5-4039-99c1-87df1a44df92","Type":"ContainerStarted","Data":"e20670e4135f223126dc69dcfd4f6a07105d4c3dbc800d442890d58c5f290a72"}
Jul 24 17:42:44.010698 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:44.010599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:42:44.024212 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:44.024170    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx" podStartSLOduration=1.024156385 podStartE2EDuration="1.024156385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:44.023535736 +0000 UTC m=+2367.854788414" watchObservedRunningTime="2026-07-24 17:42:44.024156385 +0000 UTC m=+2367.855409050"
Jul 24 17:42:44.040368 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:44.040322    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq" podStartSLOduration=1.040306468 podStartE2EDuration="1.040306468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:44.037521458 +0000 UTC m=+2367.868774146" watchObservedRunningTime="2026-07-24 17:42:44.040306468 +0000 UTC m=+2367.871559157"
Jul 24 17:42:46.643900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:46.643869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:46.976561 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:46.976523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:47.359663 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:47.359567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:47.473814 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:47.473786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:50.182601 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:50.182560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:50.338730 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:50.338697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:50.462249 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:50.462097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:50.651439 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:50.651377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:50.883596 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:50.883522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:50.982170 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:50.982134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:51.010841 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.010805    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"]
Jul 24 17:42:51.014180 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.014161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:42:51.016381 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.016362    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-368755\""
Jul 24 17:42:51.016588 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.016574    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-368755-dockercfg-cd79h\""
Jul 24 17:42:51.025676 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.025626    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"]
Jul 24 17:42:51.061940 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.061912    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"]
Jul 24 17:42:51.065334 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.065320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:42:51.073347 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.073328    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"]
Jul 24 17:42:51.164256 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.164183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-368755-7986c657d4-cst9m\" (UID: \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:42:51.164256 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.164237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66n42\" (UniqueName: \"kubernetes.io/projected/af39f2f3-fa01-47ad-a940-e9330a4a6647-kube-api-access-66n42\") pod \"payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd\" (UID: \"af39f2f3-fa01-47ad-a940-e9330a4a6647\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:42:51.164462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.164319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b59mj\" (UniqueName: \"kubernetes.io/projected/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-kube-api-access-b59mj\") pod \"payload-processing-e2e-inf-b-368755-7986c657d4-cst9m\" (UID: \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:42:51.164462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.164347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f2f3-fa01-47ad-a940-e9330a4a6647-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd\" (UID: \"af39f2f3-fa01-47ad-a940-e9330a4a6647\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:42:51.264926 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.264895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66n42\" (UniqueName: \"kubernetes.io/projected/af39f2f3-fa01-47ad-a940-e9330a4a6647-kube-api-access-66n42\") pod \"payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd\" (UID: \"af39f2f3-fa01-47ad-a940-e9330a4a6647\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:42:51.265354 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.264952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b59mj\" (UniqueName: \"kubernetes.io/projected/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-kube-api-access-b59mj\") pod \"payload-processing-e2e-inf-b-368755-7986c657d4-cst9m\" (UID: \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:42:51.265354 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.264979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f2f3-fa01-47ad-a940-e9330a4a6647-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd\" (UID: \"af39f2f3-fa01-47ad-a940-e9330a4a6647\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:42:51.265354 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.265048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-368755-7986c657d4-cst9m\" (UID: \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:42:51.265788 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.265768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f2f3-fa01-47ad-a940-e9330a4a6647-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd\" (UID: \"af39f2f3-fa01-47ad-a940-e9330a4a6647\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:42:51.265868 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.265802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-368755-7986c657d4-cst9m\" (UID: \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:42:51.273033 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.273004    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66n42\" (UniqueName: \"kubernetes.io/projected/af39f2f3-fa01-47ad-a940-e9330a4a6647-kube-api-access-66n42\") pod \"payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd\" (UID: \"af39f2f3-fa01-47ad-a940-e9330a4a6647\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:42:51.273141 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.273004    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b59mj\" (UniqueName: \"kubernetes.io/projected/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-kube-api-access-b59mj\") pod \"payload-processing-e2e-inf-b-368755-7986c657d4-cst9m\" (UID: \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:42:51.327184 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.327163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:42:51.375528 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.375500    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:42:51.678534 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.678507    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"]
Jul 24 17:42:51.680911 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:42:51.680885    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41dc5f8_8e25_4a5b_bbd1_3b5c315eaf47.slice/crio-4f5668799f8c3d418e2984a15fb66151acd21c5cdad3fec1683ed47eaea11bc4 WatchSource:0}: Error finding container 4f5668799f8c3d418e2984a15fb66151acd21c5cdad3fec1683ed47eaea11bc4: Status 404 returned error can't find the container with id 4f5668799f8c3d418e2984a15fb66151acd21c5cdad3fec1683ed47eaea11bc4
Jul 24 17:42:51.739189 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:51.739163    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"]
Jul 24 17:42:51.740902 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:42:51.740874    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf39f2f3_fa01_47ad_a940_e9330a4a6647.slice/crio-f1fbffa7b4585c2a16b2b8566284499bbc852d06e7f7313f7b906192691b2e7f WatchSource:0}: Error finding container f1fbffa7b4585c2a16b2b8566284499bbc852d06e7f7313f7b906192691b2e7f: Status 404 returned error can't find the container with id f1fbffa7b4585c2a16b2b8566284499bbc852d06e7f7313f7b906192691b2e7f
Jul 24 17:42:52.048180 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:52.048138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m" event={"ID":"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47","Type":"ContainerStarted","Data":"e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077"}
Jul 24 17:42:52.048374 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:52.048306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:42:52.048374 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:52.048337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m" event={"ID":"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47","Type":"ContainerStarted","Data":"4f5668799f8c3d418e2984a15fb66151acd21c5cdad3fec1683ed47eaea11bc4"}
Jul 24 17:42:52.049805 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:52.049775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd" event={"ID":"af39f2f3-fa01-47ad-a940-e9330a4a6647","Type":"ContainerStarted","Data":"0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87"}
Jul 24 17:42:52.049901 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:52.049816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd" event={"ID":"af39f2f3-fa01-47ad-a940-e9330a4a6647","Type":"ContainerStarted","Data":"f1fbffa7b4585c2a16b2b8566284499bbc852d06e7f7313f7b906192691b2e7f"}
Jul 24 17:42:52.049901 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:52.049835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:42:52.065852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:52.065812    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m" podStartSLOduration=2.065797372 podStartE2EDuration="2.065797372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:52.0641804 +0000 UTC m=+2375.895433099" watchObservedRunningTime="2026-07-24 17:42:52.065797372 +0000 UTC m=+2375.897050058"
Jul 24 17:42:52.081623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:52.081583    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd" podStartSLOduration=1.081570901 podStartE2EDuration="1.081570901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:52.080145351 +0000 UTC m=+2375.911398041" watchObservedRunningTime="2026-07-24 17:42:52.081570901 +0000 UTC m=+2375.912823587"
Jul 24 17:42:54.719917 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:54.719887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:42:55.018367 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:55.018280    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq" podUID="f5150981-d8d5-4039-99c1-87df1a44df92" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.45:9004: connect: connection refused"
Jul 24 17:42:55.018620 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:42:55.018601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:43:01.250219 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.250177    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"]
Jul 24 17:43:01.254011 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.253988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.256469 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.256449    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5e8b71\"/\"openshift-service-ca.crt\""
Jul 24 17:43:01.257189 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.257168    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5e8b71\"/\"default-dockercfg-5jwpj\""
Jul 24 17:43:01.257242 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.257217    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5e8b71\"/\"kube-root-ca.crt\""
Jul 24 17:43:01.257307 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.257219    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5e8b71\"/\"test-model-5e8b71-kserve-self-signed-certs\""
Jul 24 17:43:01.262444 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.262425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"]
Jul 24 17:43:01.348985 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.348957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:01.355183 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.355162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-dshm\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.355260 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.355190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-tmp-dir\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.355260 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.355210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvx5t\" (UniqueName: \"kubernetes.io/projected/5a445d69-6786-4dcf-943e-11307097627f-kube-api-access-dvx5t\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.355260 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.355227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a445d69-6786-4dcf-943e-11307097627f-tls-certs\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.355353 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.355339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-home\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.355387 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.355362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-model-cache\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.456382 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.456345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-dshm\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.456546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.456386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-tmp-dir\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.456546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.456410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvx5t\" (UniqueName: \"kubernetes.io/projected/5a445d69-6786-4dcf-943e-11307097627f-kube-api-access-dvx5t\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.456546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.456530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a445d69-6786-4dcf-943e-11307097627f-tls-certs\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.456759 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.456622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-home\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.456759 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.456673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-model-cache\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.456876 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.456796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-tmp-dir\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.456930 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.456914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-home\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.457006 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.456987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-model-cache\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.458827 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.458808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-dshm\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.459084 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.459069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a445d69-6786-4dcf-943e-11307097627f-tls-certs\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.464956 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.464938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvx5t\" (UniqueName: \"kubernetes.io/projected/5a445d69-6786-4dcf-943e-11307097627f-kube-api-access-dvx5t\") pod \"test-model-5e8b71-kserve-57bf95798f-px5qf\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") " pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.564969 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.564899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:01.698187 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:01.698164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"]
Jul 24 17:43:01.699935 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:43:01.699908    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a445d69_6786_4dcf_943e_11307097627f.slice/crio-ce2f392c23394de2f77bac13b054185e57331a4f3646e8c08d52df764d212a34 WatchSource:0}: Error finding container ce2f392c23394de2f77bac13b054185e57331a4f3646e8c08d52df764d212a34: Status 404 returned error can't find the container with id ce2f392c23394de2f77bac13b054185e57331a4f3646e8c08d52df764d212a34
Jul 24 17:43:02.097581 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:02.097547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf" event={"ID":"5a445d69-6786-4dcf-943e-11307097627f","Type":"ContainerStarted","Data":"36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc"}
Jul 24 17:43:02.097792 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:02.097591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf" event={"ID":"5a445d69-6786-4dcf-943e-11307097627f","Type":"ContainerStarted","Data":"ce2f392c23394de2f77bac13b054185e57331a4f3646e8c08d52df764d212a34"}
Jul 24 17:43:02.116740 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:02.116691    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf" podStartSLOduration=1.116675923 podStartE2EDuration="1.116675923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:02.114604099 +0000 UTC m=+2385.945856785" watchObservedRunningTime="2026-07-24 17:43:02.116675923 +0000 UTC m=+2385.947928669"
Jul 24 17:43:03.056316 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:03.056281    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:43:03.056796 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:03.056394    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:43:05.017934 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:05.017904    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:43:11.565283 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:11.565244    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:11.565747 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:11.565298    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:11.577869 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:11.577850    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:12.158411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:12.158385    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:43:16.760574 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:16.760548    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:43:16.765754 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:16.765733    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:43:40.484288 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:40.484169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:43:42.729097 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:43:42.729064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:20.109822 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:20.109789    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"]
Jul 24 17:44:20.135872 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:20.135845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:21.093191 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:21.093148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:27.716710 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:27.716676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:31.861813 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:31.861779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:31.894383 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:31.894353    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"]
Jul 24 17:44:31.894623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:31.894601    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx" podUID="e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1" containerName="payload-processing" containerID="cri-o://629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd" gracePeriod=30
Jul 24 17:44:31.902089 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:31.902060    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"]
Jul 24 17:44:31.902325 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:31.902300    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq" podUID="f5150981-d8d5-4039-99c1-87df1a44df92" containerName="payload-pre-processing" containerID="cri-o://bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b" gracePeriod=30
Jul 24 17:44:31.962293 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:31.962265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:32.061365 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.061334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:32.148769 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.148743    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:44:32.176286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.176250    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:44:32.269578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.269551    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5150981-d8d5-4039-99c1-87df1a44df92-plugins-config-volume\") pod \"f5150981-d8d5-4039-99c1-87df1a44df92\" (UID: \"f5150981-d8d5-4039-99c1-87df1a44df92\") "
Jul 24 17:44:32.269578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.269583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpgfr\" (UniqueName: \"kubernetes.io/projected/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-kube-api-access-vpgfr\") pod \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\" (UID: \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\") "
Jul 24 17:44:32.269817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.269631    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-plugins-config-volume\") pod \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\" (UID: \"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1\") "
Jul 24 17:44:32.269817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.269752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ftf6\" (UniqueName: \"kubernetes.io/projected/f5150981-d8d5-4039-99c1-87df1a44df92-kube-api-access-9ftf6\") pod \"f5150981-d8d5-4039-99c1-87df1a44df92\" (UID: \"f5150981-d8d5-4039-99c1-87df1a44df92\") "
Jul 24 17:44:32.270021 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.269984    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1" (UID: "e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:44:32.270137 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.270016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5150981-d8d5-4039-99c1-87df1a44df92-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5150981-d8d5-4039-99c1-87df1a44df92" (UID: "f5150981-d8d5-4039-99c1-87df1a44df92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:44:32.270137 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.270107    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5150981-d8d5-4039-99c1-87df1a44df92-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:32.270137 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.270120    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:32.271883 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.271855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5150981-d8d5-4039-99c1-87df1a44df92-kube-api-access-9ftf6" (OuterVolumeSpecName: "kube-api-access-9ftf6") pod "f5150981-d8d5-4039-99c1-87df1a44df92" (UID: "f5150981-d8d5-4039-99c1-87df1a44df92"). InnerVolumeSpecName "kube-api-access-9ftf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:44:32.271955 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.271930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-kube-api-access-vpgfr" (OuterVolumeSpecName: "kube-api-access-vpgfr") pod "e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1" (UID: "e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1"). InnerVolumeSpecName "kube-api-access-vpgfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:44:32.371335 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.371302    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpgfr\" (UniqueName: \"kubernetes.io/projected/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1-kube-api-access-vpgfr\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:32.371335 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.371330    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ftf6\" (UniqueName: \"kubernetes.io/projected/f5150981-d8d5-4039-99c1-87df1a44df92-kube-api-access-9ftf6\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:32.522272 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.522239    2563 generic.go:356] "Generic (PLEG): container finished" podID="f5150981-d8d5-4039-99c1-87df1a44df92" containerID="bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b" exitCode=0
Jul 24 17:44:32.522484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.522294    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"
Jul 24 17:44:32.522484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.522325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq" event={"ID":"f5150981-d8d5-4039-99c1-87df1a44df92","Type":"ContainerDied","Data":"bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b"}
Jul 24 17:44:32.522484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.522380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq" event={"ID":"f5150981-d8d5-4039-99c1-87df1a44df92","Type":"ContainerDied","Data":"e20670e4135f223126dc69dcfd4f6a07105d4c3dbc800d442890d58c5f290a72"}
Jul 24 17:44:32.522484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.522416    2563 scope.go:117] "RemoveContainer" containerID="bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b"
Jul 24 17:44:32.523852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.523801    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1" containerID="629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd" exitCode=0
Jul 24 17:44:32.523852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.523825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx" event={"ID":"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1","Type":"ContainerDied","Data":"629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd"}
Jul 24 17:44:32.523852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.523849    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"
Jul 24 17:44:32.524042 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.523859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx" event={"ID":"e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1","Type":"ContainerDied","Data":"9d6c60de9ec9ca26571cc3ebe1033bbfb44dab42f5a37174f4c9b6a04bd72d6d"}
Jul 24 17:44:32.533025 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.533010    2563 scope.go:117] "RemoveContainer" containerID="bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b"
Jul 24 17:44:32.533284 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:44:32.533260    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b\": container with ID starting with bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b not found: ID does not exist" containerID="bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b"
Jul 24 17:44:32.533392 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.533293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b"} err="failed to get container status \"bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b\": rpc error: code = NotFound desc = could not find container \"bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b\": container with ID starting with bf01390cb687d663e650dc90dce7796bfb6009e1210e9d90c15fc5cd6ddbd41b not found: ID does not exist"
Jul 24 17:44:32.533392 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.533314    2563 scope.go:117] "RemoveContainer" containerID="629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd"
Jul 24 17:44:32.540944 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.540923    2563 scope.go:117] "RemoveContainer" containerID="629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd"
Jul 24 17:44:32.541201 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:44:32.541181    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd\": container with ID starting with 629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd not found: ID does not exist" containerID="629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd"
Jul 24 17:44:32.541254 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.541209    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd"} err="failed to get container status \"629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd\": rpc error: code = NotFound desc = could not find container \"629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd\": container with ID starting with 629c925ba2fb5ba1f35dd8ced78b7eabf4345873be73ccc677c0d9d5bff1d2dd not found: ID does not exist"
Jul 24 17:44:32.544222 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.544202    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"]
Jul 24 17:44:32.547705 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.547686    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5e8b71-5fdcc7f66d-znkqq"]
Jul 24 17:44:32.557759 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.557736    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"]
Jul 24 17:44:32.562024 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.562003    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5e8b71-5c9587c4d6-p5vtx"]
Jul 24 17:44:32.688380 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.688351    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1" path="/var/lib/kubelet/pods/e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1/volumes"
Jul 24 17:44:32.688729 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:32.688714    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5150981-d8d5-4039-99c1-87df1a44df92" path="/var/lib/kubelet/pods/f5150981-d8d5-4039-99c1-87df1a44df92/volumes"
Jul 24 17:44:35.129247 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.129190    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf" podUID="5a445d69-6786-4dcf-943e-11307097627f" containerName="main" containerID="cri-o://36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc" gracePeriod=45
Jul 24 17:44:35.374145 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.374124    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:44:35.498090 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.498069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a445d69-6786-4dcf-943e-11307097627f-tls-certs\") pod \"5a445d69-6786-4dcf-943e-11307097627f\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") "
Jul 24 17:44:35.498241 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.498135    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-home\") pod \"5a445d69-6786-4dcf-943e-11307097627f\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") "
Jul 24 17:44:35.498241 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.498156    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-tmp-dir\") pod \"5a445d69-6786-4dcf-943e-11307097627f\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") "
Jul 24 17:44:35.498241 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.498187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvx5t\" (UniqueName: \"kubernetes.io/projected/5a445d69-6786-4dcf-943e-11307097627f-kube-api-access-dvx5t\") pod \"5a445d69-6786-4dcf-943e-11307097627f\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") "
Jul 24 17:44:35.498241 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.498215    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-dshm\") pod \"5a445d69-6786-4dcf-943e-11307097627f\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") "
Jul 24 17:44:35.498241 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.498230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-model-cache\") pod \"5a445d69-6786-4dcf-943e-11307097627f\" (UID: \"5a445d69-6786-4dcf-943e-11307097627f\") "
Jul 24 17:44:35.498510 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.498483    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-home" (OuterVolumeSpecName: "home") pod "5a445d69-6786-4dcf-943e-11307097627f" (UID: "5a445d69-6786-4dcf-943e-11307097627f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:44:35.498570 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.498495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a445d69-6786-4dcf-943e-11307097627f" (UID: "5a445d69-6786-4dcf-943e-11307097627f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:44:35.498621 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.498565    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a445d69-6786-4dcf-943e-11307097627f" (UID: "5a445d69-6786-4dcf-943e-11307097627f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:44:35.500727 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.500698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-dshm" (OuterVolumeSpecName: "dshm") pod "5a445d69-6786-4dcf-943e-11307097627f" (UID: "5a445d69-6786-4dcf-943e-11307097627f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:44:35.500890 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.500823    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a445d69-6786-4dcf-943e-11307097627f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a445d69-6786-4dcf-943e-11307097627f" (UID: "5a445d69-6786-4dcf-943e-11307097627f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:44:35.500890 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.500851    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a445d69-6786-4dcf-943e-11307097627f-kube-api-access-dvx5t" (OuterVolumeSpecName: "kube-api-access-dvx5t") pod "5a445d69-6786-4dcf-943e-11307097627f" (UID: "5a445d69-6786-4dcf-943e-11307097627f"). InnerVolumeSpecName "kube-api-access-dvx5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:44:35.547133 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.547107    2563 generic.go:356] "Generic (PLEG): container finished" podID="5a445d69-6786-4dcf-943e-11307097627f" containerID="36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc" exitCode=0
Jul 24 17:44:35.547214 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.547155    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"
Jul 24 17:44:35.547214 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.547194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf" event={"ID":"5a445d69-6786-4dcf-943e-11307097627f","Type":"ContainerDied","Data":"36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc"}
Jul 24 17:44:35.547292 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.547238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf" event={"ID":"5a445d69-6786-4dcf-943e-11307097627f","Type":"ContainerDied","Data":"ce2f392c23394de2f77bac13b054185e57331a4f3646e8c08d52df764d212a34"}
Jul 24 17:44:35.547292 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.547256    2563 scope.go:117] "RemoveContainer" containerID="36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc"
Jul 24 17:44:35.556035 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.556018    2563 scope.go:117] "RemoveContainer" containerID="36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc"
Jul 24 17:44:35.556304 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:44:35.556286    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc\": container with ID starting with 36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc not found: ID does not exist" containerID="36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc"
Jul 24 17:44:35.556360 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.556312    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc"} err="failed to get container status \"36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc\": rpc error: code = NotFound desc = could not find container \"36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc\": container with ID starting with 36a4582d6c91db5b70a7bb87b07ef5d1cbbee651c4ce2fb3a750e29b82a455bc not found: ID does not exist"
Jul 24 17:44:35.568828 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.568808    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"]
Jul 24 17:44:35.572401 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.572380    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5e8b71/test-model-5e8b71-kserve-57bf95798f-px5qf"]
Jul 24 17:44:35.599003 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.598980    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-home\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:35.599075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.599017    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-tmp-dir\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:35.599075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.599027    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvx5t\" (UniqueName: \"kubernetes.io/projected/5a445d69-6786-4dcf-943e-11307097627f-kube-api-access-dvx5t\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:35.599075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.599035    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-dshm\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:35.599075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.599044    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a445d69-6786-4dcf-943e-11307097627f-model-cache\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:35.599075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:35.599052    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a445d69-6786-4dcf-943e-11307097627f-tls-certs\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:36.281397 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:36.281355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:36.688966 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:36.688884    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a445d69-6786-4dcf-943e-11307097627f" path="/var/lib/kubelet/pods/5a445d69-6786-4dcf-943e-11307097627f/volumes"
Jul 24 17:44:36.819008 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:36.818974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:42.777705 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:42.777674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:42.846107 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:42.846076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:43.070518 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:43.070434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:43.288803 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:43.288774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:54.009100 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.009063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:54.036338 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.036303    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"]
Jul 24 17:44:54.036609 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.036583    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m" podUID="a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47" containerName="payload-processing" containerID="cri-o://e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077" gracePeriod=30
Jul 24 17:44:54.042462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.042435    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"]
Jul 24 17:44:54.042694 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.042630    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd" podUID="af39f2f3-fa01-47ad-a940-e9330a4a6647" containerName="payload-pre-processing" containerID="cri-o://0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87" gracePeriod=30
Jul 24 17:44:54.109242 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.109206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:54.210538 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.210509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:54.312438 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.312418    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:44:54.315077 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.315057    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:44:54.354391 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.354351    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66n42\" (UniqueName: \"kubernetes.io/projected/af39f2f3-fa01-47ad-a940-e9330a4a6647-kube-api-access-66n42\") pod \"af39f2f3-fa01-47ad-a940-e9330a4a6647\" (UID: \"af39f2f3-fa01-47ad-a940-e9330a4a6647\") "
Jul 24 17:44:54.354583 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.354468    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-plugins-config-volume\") pod \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\" (UID: \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\") "
Jul 24 17:44:54.354583 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.354499    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f2f3-fa01-47ad-a940-e9330a4a6647-plugins-config-volume\") pod \"af39f2f3-fa01-47ad-a940-e9330a4a6647\" (UID: \"af39f2f3-fa01-47ad-a940-e9330a4a6647\") "
Jul 24 17:44:54.354583 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.354542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b59mj\" (UniqueName: \"kubernetes.io/projected/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-kube-api-access-b59mj\") pod \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\" (UID: \"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47\") "
Jul 24 17:44:54.354953 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.354927    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47" (UID: "a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:44:54.354953 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.354946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af39f2f3-fa01-47ad-a940-e9330a4a6647-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af39f2f3-fa01-47ad-a940-e9330a4a6647" (UID: "af39f2f3-fa01-47ad-a940-e9330a4a6647"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:44:54.355096 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.355070    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f2f3-fa01-47ad-a940-e9330a4a6647-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:54.355096 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.355089    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:54.356765 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.356741    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af39f2f3-fa01-47ad-a940-e9330a4a6647-kube-api-access-66n42" (OuterVolumeSpecName: "kube-api-access-66n42") pod "af39f2f3-fa01-47ad-a940-e9330a4a6647" (UID: "af39f2f3-fa01-47ad-a940-e9330a4a6647"). InnerVolumeSpecName "kube-api-access-66n42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:44:54.357146 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.357125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-kube-api-access-b59mj" (OuterVolumeSpecName: "kube-api-access-b59mj") pod "a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47" (UID: "a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47"). InnerVolumeSpecName "kube-api-access-b59mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:44:54.455923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.455881    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b59mj\" (UniqueName: \"kubernetes.io/projected/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47-kube-api-access-b59mj\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:54.455923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.455913    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66n42\" (UniqueName: \"kubernetes.io/projected/af39f2f3-fa01-47ad-a940-e9330a4a6647-kube-api-access-66n42\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:44:54.636014 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.635925    2563 generic.go:356] "Generic (PLEG): container finished" podID="af39f2f3-fa01-47ad-a940-e9330a4a6647" containerID="0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87" exitCode=0
Jul 24 17:44:54.636014 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.635974    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"
Jul 24 17:44:54.636216 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.635966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd" event={"ID":"af39f2f3-fa01-47ad-a940-e9330a4a6647","Type":"ContainerDied","Data":"0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87"}
Jul 24 17:44:54.636216 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.636119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd" event={"ID":"af39f2f3-fa01-47ad-a940-e9330a4a6647","Type":"ContainerDied","Data":"f1fbffa7b4585c2a16b2b8566284499bbc852d06e7f7313f7b906192691b2e7f"}
Jul 24 17:44:54.636216 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.636152    2563 scope.go:117] "RemoveContainer" containerID="0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87"
Jul 24 17:44:54.637378 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.637354    2563 generic.go:356] "Generic (PLEG): container finished" podID="a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47" containerID="e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077" exitCode=0
Jul 24 17:44:54.637378 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.637372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m" event={"ID":"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47","Type":"ContainerDied","Data":"e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077"}
Jul 24 17:44:54.637567 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.637394    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"
Jul 24 17:44:54.637567 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.637414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m" event={"ID":"a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47","Type":"ContainerDied","Data":"4f5668799f8c3d418e2984a15fb66151acd21c5cdad3fec1683ed47eaea11bc4"}
Jul 24 17:44:54.645261 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.645237    2563 scope.go:117] "RemoveContainer" containerID="0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87"
Jul 24 17:44:54.645535 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:44:54.645508    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87\": container with ID starting with 0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87 not found: ID does not exist" containerID="0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87"
Jul 24 17:44:54.645609 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.645542    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87"} err="failed to get container status \"0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87\": rpc error: code = NotFound desc = could not find container \"0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87\": container with ID starting with 0f4731a049b33092a442497583f0bbf0f9328d004eb94078b114ce2d57939c87 not found: ID does not exist"
Jul 24 17:44:54.645609 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.645558    2563 scope.go:117] "RemoveContainer" containerID="e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077"
Jul 24 17:44:54.656330 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.656314    2563 scope.go:117] "RemoveContainer" containerID="e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077"
Jul 24 17:44:54.656590 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:44:54.656567    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077\": container with ID starting with e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077 not found: ID does not exist" containerID="e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077"
Jul 24 17:44:54.656697 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.656596    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077"} err="failed to get container status \"e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077\": rpc error: code = NotFound desc = could not find container \"e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077\": container with ID starting with e1d77558775e19b3edf24f28c6d52f023bbd1d40d74254bcf9c247ef88412077 not found: ID does not exist"
Jul 24 17:44:54.687926 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.687898    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"]
Jul 24 17:44:54.696028 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.696005    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-368755-7986c657d4-cst9m"]
Jul 24 17:44:54.712841 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.712816    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"]
Jul 24 17:44:54.719485 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:54.719462    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-368755-55974c9cfb-m2qsd"]
Jul 24 17:44:56.687623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:56.687577    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47" path="/var/lib/kubelet/pods/a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47/volumes"
Jul 24 17:44:56.688132 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:56.688097    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af39f2f3-fa01-47ad-a940-e9330a4a6647" path="/var/lib/kubelet/pods/af39f2f3-fa01-47ad-a940-e9330a4a6647/volumes"
Jul 24 17:44:58.523973 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:58.523928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:44:58.900880 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:44:58.900807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:00.139968 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.139928    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh"]
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140283    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1" containerName="payload-processing"
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140293    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1" containerName="payload-processing"
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140301    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a445d69-6786-4dcf-943e-11307097627f" containerName="main"
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140307    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a445d69-6786-4dcf-943e-11307097627f" containerName="main"
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140319    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5150981-d8d5-4039-99c1-87df1a44df92" containerName="payload-pre-processing"
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140327    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5150981-d8d5-4039-99c1-87df1a44df92" containerName="payload-pre-processing"
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140336    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47" containerName="payload-processing"
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140341    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47" containerName="payload-processing"
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140352    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af39f2f3-fa01-47ad-a940-e9330a4a6647" containerName="payload-pre-processing"
Jul 24 17:45:00.140364 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140357    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af39f2f3-fa01-47ad-a940-e9330a4a6647" containerName="payload-pre-processing"
Jul 24 17:45:00.140751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140404    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41dc5f8-8e25-4a5b-bbd1-3b5c315eaf47" containerName="payload-processing"
Jul 24 17:45:00.140751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140412    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ba02b8-f3b0-4f45-95ee-2707a19ef3a1" containerName="payload-processing"
Jul 24 17:45:00.140751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140417    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a445d69-6786-4dcf-943e-11307097627f" containerName="main"
Jul 24 17:45:00.140751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140425    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5150981-d8d5-4039-99c1-87df1a44df92" containerName="payload-pre-processing"
Jul 24 17:45:00.140751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.140430    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af39f2f3-fa01-47ad-a940-e9330a4a6647" containerName="payload-pre-processing"
Jul 24 17:45:00.144959 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.144940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh"
Jul 24 17:45:00.147857 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.147830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8llqd\""
Jul 24 17:45:00.157795 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.157769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh"]
Jul 24 17:45:00.161371 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.161352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh"
Jul 24 17:45:00.292209 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.292171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh"]
Jul 24 17:45:00.296389 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:45:00.296361    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d23dc_8247_49e3_9cf0_0a0e2a35e033.slice/crio-de308109ce21c934c1887a387f4f8aedcacfc1685b76b85903ed0d2e628d0beb WatchSource:0}: Error finding container de308109ce21c934c1887a387f4f8aedcacfc1685b76b85903ed0d2e628d0beb: Status 404 returned error can't find the container with id de308109ce21c934c1887a387f4f8aedcacfc1685b76b85903ed0d2e628d0beb
Jul 24 17:45:00.298200 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.298180    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:45:00.671192 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:00.671154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh" event={"ID":"536d23dc-8247-49e3-9cf0-0a0e2a35e033","Type":"ContainerStarted","Data":"de308109ce21c934c1887a387f4f8aedcacfc1685b76b85903ed0d2e628d0beb"}
Jul 24 17:45:04.692185 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:04.692146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh" event={"ID":"536d23dc-8247-49e3-9cf0-0a0e2a35e033","Type":"ContainerStarted","Data":"e5336c768a88cc444aff19c202cae982522e4c015a8919384e92ea92224ceac7"}
Jul 24 17:45:04.706251 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:04.706202    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh" podStartSLOduration=1.223986072 podStartE2EDuration="4.706182724s" totalImagesPullingTime="3.482196652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:04.705886487 +0000 UTC m=+2508.537139174" watchObservedRunningTime="2026-07-24 17:45:04.706182724 +0000 UTC m=+2508.537435412"
Jul 24 17:45:04.878931 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:04.878900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:04.951431 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:04.951359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:05.241515 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:05.241474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:05.460412 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:05.460383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:05.698572 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:05.698486    2563 generic.go:356] "Generic (PLEG): container finished" podID="536d23dc-8247-49e3-9cf0-0a0e2a35e033" containerID="e5336c768a88cc444aff19c202cae982522e4c015a8919384e92ea92224ceac7" exitCode=0
Jul 24 17:45:05.698572 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:05.698539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh" event={"ID":"536d23dc-8247-49e3-9cf0-0a0e2a35e033","Type":"ContainerDied","Data":"e5336c768a88cc444aff19c202cae982522e4c015a8919384e92ea92224ceac7"}
Jul 24 17:45:05.786613 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:05.786584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:06.125809 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:06.125772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:06.834766 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:06.834736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:06.846425 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:06.846406    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh"
Jul 24 17:45:07.709673 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:07.709576    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh"
Jul 24 17:45:07.709673 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:07.709590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748585-24kqh" event={"ID":"536d23dc-8247-49e3-9cf0-0a0e2a35e033","Type":"ContainerDied","Data":"de308109ce21c934c1887a387f4f8aedcacfc1685b76b85903ed0d2e628d0beb"}
Jul 24 17:45:07.709673 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:07.709632    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de308109ce21c934c1887a387f4f8aedcacfc1685b76b85903ed0d2e628d0beb"
Jul 24 17:45:08.730498 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:08.730471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:08.837269 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:08.837233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:08.923883 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:08.923856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:09.148198 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.148116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:09.441204 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.441128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:09.551439 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.551404    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"]
Jul 24 17:45:09.551813 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.551798    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536d23dc-8247-49e3-9cf0-0a0e2a35e033" containerName="cleanup"
Jul 24 17:45:09.551869 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.551815    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="536d23dc-8247-49e3-9cf0-0a0e2a35e033" containerName="cleanup"
Jul 24 17:45:09.551917 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.551897    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="536d23dc-8247-49e3-9cf0-0a0e2a35e033" containerName="cleanup"
Jul 24 17:45:09.554903 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.554887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:09.557298 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.557279    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ac84f7\""
Jul 24 17:45:09.570879 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.570849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"]
Jul 24 17:45:09.573159 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.573138    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"]
Jul 24 17:45:09.576379 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.576360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:09.578579 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.578563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ac84f7-dockercfg-dbps2\""
Jul 24 17:45:09.586084 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.586064    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"]
Jul 24 17:45:09.695449 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.695373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:09.701743 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.701717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f68b994-4a9c-49eb-a73e-8c9dab525227-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx\" (UID: \"5f68b994-4a9c-49eb-a73e-8c9dab525227\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:09.701830 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.701787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6\" (UID: \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:09.701867 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.701848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hk8\" (UniqueName: \"kubernetes.io/projected/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-kube-api-access-48hk8\") pod \"payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6\" (UID: \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:09.701915 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.701889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpc85\" (UniqueName: \"kubernetes.io/projected/5f68b994-4a9c-49eb-a73e-8c9dab525227-kube-api-access-xpc85\") pod \"payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx\" (UID: \"5f68b994-4a9c-49eb-a73e-8c9dab525227\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:09.802381 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.802350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hk8\" (UniqueName: \"kubernetes.io/projected/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-kube-api-access-48hk8\") pod \"payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6\" (UID: \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:09.802900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.802402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpc85\" (UniqueName: \"kubernetes.io/projected/5f68b994-4a9c-49eb-a73e-8c9dab525227-kube-api-access-xpc85\") pod \"payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx\" (UID: \"5f68b994-4a9c-49eb-a73e-8c9dab525227\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:09.802900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.802479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f68b994-4a9c-49eb-a73e-8c9dab525227-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx\" (UID: \"5f68b994-4a9c-49eb-a73e-8c9dab525227\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:09.802900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.802537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6\" (UID: \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:09.803335 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.803313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f68b994-4a9c-49eb-a73e-8c9dab525227-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx\" (UID: \"5f68b994-4a9c-49eb-a73e-8c9dab525227\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:09.803423 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.803360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6\" (UID: \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:09.814256 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.814232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpc85\" (UniqueName: \"kubernetes.io/projected/5f68b994-4a9c-49eb-a73e-8c9dab525227-kube-api-access-xpc85\") pod \"payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx\" (UID: \"5f68b994-4a9c-49eb-a73e-8c9dab525227\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:09.814501 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.814482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hk8\" (UniqueName: \"kubernetes.io/projected/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-kube-api-access-48hk8\") pod \"payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6\" (UID: \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:09.865900 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.865878    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:09.887031 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:09.887010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:10.028258 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.028228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"]
Jul 24 17:45:10.030436 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:45:10.030411    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f68b994_4a9c_49eb_a73e_8c9dab525227.slice/crio-16f7fdf3afab3db6906de529dba7f3e3aad74e05b45cfaf541532a1b161dcf64 WatchSource:0}: Error finding container 16f7fdf3afab3db6906de529dba7f3e3aad74e05b45cfaf541532a1b161dcf64: Status 404 returned error can't find the container with id 16f7fdf3afab3db6906de529dba7f3e3aad74e05b45cfaf541532a1b161dcf64
Jul 24 17:45:10.047694 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.047677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"]
Jul 24 17:45:10.049845 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:45:10.049822    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3718d43b_1db5_4b19_b4d8_ab1c415ba5d8.slice/crio-7727b46fa1016044501da2ee4b2b0ffa68a53c5bf31854f3cc2d37f61a84b971 WatchSource:0}: Error finding container 7727b46fa1016044501da2ee4b2b0ffa68a53c5bf31854f3cc2d37f61a84b971: Status 404 returned error can't find the container with id 7727b46fa1016044501da2ee4b2b0ffa68a53c5bf31854f3cc2d37f61a84b971
Jul 24 17:45:10.726110 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.726071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6" event={"ID":"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8","Type":"ContainerStarted","Data":"cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c"}
Jul 24 17:45:10.726322 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.726134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6" event={"ID":"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8","Type":"ContainerStarted","Data":"7727b46fa1016044501da2ee4b2b0ffa68a53c5bf31854f3cc2d37f61a84b971"}
Jul 24 17:45:10.726322 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.726189    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:10.727633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.727607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx" event={"ID":"5f68b994-4a9c-49eb-a73e-8c9dab525227","Type":"ContainerStarted","Data":"d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585"}
Jul 24 17:45:10.727768 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.727673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx" event={"ID":"5f68b994-4a9c-49eb-a73e-8c9dab525227","Type":"ContainerStarted","Data":"16f7fdf3afab3db6906de529dba7f3e3aad74e05b45cfaf541532a1b161dcf64"}
Jul 24 17:45:10.727768 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.727713    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:10.744655 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.744606    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6" podStartSLOduration=1.74459415 podStartE2EDuration="1.74459415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:10.744052175 +0000 UTC m=+2514.575304863" watchObservedRunningTime="2026-07-24 17:45:10.74459415 +0000 UTC m=+2514.575846890"
Jul 24 17:45:10.761383 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:10.761332    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx" podStartSLOduration=1.761319595 podStartE2EDuration="1.761319595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:10.759642608 +0000 UTC m=+2514.590895296" watchObservedRunningTime="2026-07-24 17:45:10.761319595 +0000 UTC m=+2514.592572282"
Jul 24 17:45:13.188668 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:13.188614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:13.840972 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:13.840935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:14.147079 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:14.146999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:14.515896 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:14.515866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:16.877049 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:16.877024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:16.961613 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:16.961576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:17.070817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.070785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:17.392158 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.392102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:17.666167 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.666082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:17.769507 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.769472    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"]
Jul 24 17:45:17.771625 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.771604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:17.774214 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.774199    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:17.776408 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.776391    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-94ab01-dockercfg-w5mhn\""
Jul 24 17:45:17.776484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.776419    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-94ab01\""
Jul 24 17:45:17.783669 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.783632    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"]
Jul 24 17:45:17.802223 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.802200    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"]
Jul 24 17:45:17.805510 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.805495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:17.818015 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.817989    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"]
Jul 24 17:45:17.880999 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.880965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade6270-04cc-401e-b168-61edb5d2e5a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c\" (UID: \"4ade6270-04cc-401e-b168-61edb5d2e5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:17.881365 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.881022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc59n\" (UniqueName: \"kubernetes.io/projected/4ade6270-04cc-401e-b168-61edb5d2e5a1-kube-api-access-hc59n\") pod \"payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c\" (UID: \"4ade6270-04cc-401e-b168-61edb5d2e5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:17.982298 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.982266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc59n\" (UniqueName: \"kubernetes.io/projected/4ade6270-04cc-401e-b168-61edb5d2e5a1-kube-api-access-hc59n\") pod \"payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c\" (UID: \"4ade6270-04cc-401e-b168-61edb5d2e5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:17.982478 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.982351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkjs\" (UniqueName: \"kubernetes.io/projected/f6c901ec-3d92-4d09-a538-fd51a74cb591-kube-api-access-jtkjs\") pod \"payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr\" (UID: \"f6c901ec-3d92-4d09-a538-fd51a74cb591\") " pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:17.982478 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.982376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c901ec-3d92-4d09-a538-fd51a74cb591-plugins-config-volume\") pod \"payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr\" (UID: \"f6c901ec-3d92-4d09-a538-fd51a74cb591\") " pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:17.982478 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.982401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade6270-04cc-401e-b168-61edb5d2e5a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c\" (UID: \"4ade6270-04cc-401e-b168-61edb5d2e5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:17.983082 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.983065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade6270-04cc-401e-b168-61edb5d2e5a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c\" (UID: \"4ade6270-04cc-401e-b168-61edb5d2e5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:17.991068 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:17.991047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc59n\" (UniqueName: \"kubernetes.io/projected/4ade6270-04cc-401e-b168-61edb5d2e5a1-kube-api-access-hc59n\") pod \"payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c\" (UID: \"4ade6270-04cc-401e-b168-61edb5d2e5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:18.082913 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.082880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkjs\" (UniqueName: \"kubernetes.io/projected/f6c901ec-3d92-4d09-a538-fd51a74cb591-kube-api-access-jtkjs\") pod \"payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr\" (UID: \"f6c901ec-3d92-4d09-a538-fd51a74cb591\") " pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:18.082913 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.082919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c901ec-3d92-4d09-a538-fd51a74cb591-plugins-config-volume\") pod \"payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr\" (UID: \"f6c901ec-3d92-4d09-a538-fd51a74cb591\") " pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:18.083600 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.083582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c901ec-3d92-4d09-a538-fd51a74cb591-plugins-config-volume\") pod \"payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr\" (UID: \"f6c901ec-3d92-4d09-a538-fd51a74cb591\") " pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:18.084769 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.084750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:18.095847 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.095609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkjs\" (UniqueName: \"kubernetes.io/projected/f6c901ec-3d92-4d09-a538-fd51a74cb591-kube-api-access-jtkjs\") pod \"payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr\" (UID: \"f6c901ec-3d92-4d09-a538-fd51a74cb591\") " pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:18.115257 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.115229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:18.271325 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.271298    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"]
Jul 24 17:45:18.273696 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:45:18.273666    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c901ec_3d92_4d09_a538_fd51a74cb591.slice/crio-c035270e6672a40faed417ddd25e288b25b0dd789fe0efde29c86f0a5e365e06 WatchSource:0}: Error finding container c035270e6672a40faed417ddd25e288b25b0dd789fe0efde29c86f0a5e365e06: Status 404 returned error can't find the container with id c035270e6672a40faed417ddd25e288b25b0dd789fe0efde29c86f0a5e365e06
Jul 24 17:45:18.433265 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.433183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"]
Jul 24 17:45:18.435446 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:45:18.435420    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ade6270_04cc_401e_b168_61edb5d2e5a1.slice/crio-69021cc89c9da37360773cc9b853f6aadedb964ece6bfd9e3d159a650280b7cb WatchSource:0}: Error finding container 69021cc89c9da37360773cc9b853f6aadedb964ece6bfd9e3d159a650280b7cb: Status 404 returned error can't find the container with id 69021cc89c9da37360773cc9b853f6aadedb964ece6bfd9e3d159a650280b7cb
Jul 24 17:45:18.767064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.767032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c" event={"ID":"4ade6270-04cc-401e-b168-61edb5d2e5a1","Type":"ContainerStarted","Data":"0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5"}
Jul 24 17:45:18.767282 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.767089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c" event={"ID":"4ade6270-04cc-401e-b168-61edb5d2e5a1","Type":"ContainerStarted","Data":"69021cc89c9da37360773cc9b853f6aadedb964ece6bfd9e3d159a650280b7cb"}
Jul 24 17:45:18.767282 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.767123    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:18.768585 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.768561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr" event={"ID":"f6c901ec-3d92-4d09-a538-fd51a74cb591","Type":"ContainerStarted","Data":"518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a"}
Jul 24 17:45:18.768725 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.768599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr" event={"ID":"f6c901ec-3d92-4d09-a538-fd51a74cb591","Type":"ContainerStarted","Data":"c035270e6672a40faed417ddd25e288b25b0dd789fe0efde29c86f0a5e365e06"}
Jul 24 17:45:18.768725 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.768672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:18.785298 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.785260    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c" podStartSLOduration=1.7852476099999999 podStartE2EDuration="1.78524761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:18.783163364 +0000 UTC m=+2522.614416051" watchObservedRunningTime="2026-07-24 17:45:18.78524761 +0000 UTC m=+2522.616500296"
Jul 24 17:45:18.800963 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:18.800921    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr" podStartSLOduration=1.800906922 podStartE2EDuration="1.800906922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:18.799103755 +0000 UTC m=+2522.630356444" watchObservedRunningTime="2026-07-24 17:45:18.800906922 +0000 UTC m=+2522.632159609"
Jul 24 17:45:21.374890 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:21.374858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:21.735515 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:21.735487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:21.735700 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:21.735672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:29.775623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:29.775587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:45:29.776126 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:29.775716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:45:45.152027 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.151998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:45.187882 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.187852    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"]
Jul 24 17:45:45.188120 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.188090    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6" podUID="3718d43b-1db5-4b19-b4d8-ab1c415ba5d8" containerName="payload-processing" containerID="cri-o://cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c" gracePeriod=30
Jul 24 17:45:45.194432 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.194392    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"]
Jul 24 17:45:45.194620 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.194597    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx" podUID="5f68b994-4a9c-49eb-a73e-8c9dab525227" containerName="payload-pre-processing" containerID="cri-o://d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585" gracePeriod=30
Jul 24 17:45:45.228581 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.228541    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:45.327922 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.327892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:45.466093 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.466071    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:45.469078 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.469057    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:45.510525 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.510501    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f68b994-4a9c-49eb-a73e-8c9dab525227-plugins-config-volume\") pod \"5f68b994-4a9c-49eb-a73e-8c9dab525227\" (UID: \"5f68b994-4a9c-49eb-a73e-8c9dab525227\") "
Jul 24 17:45:45.510689 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.510553    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48hk8\" (UniqueName: \"kubernetes.io/projected/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-kube-api-access-48hk8\") pod \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\" (UID: \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\") "
Jul 24 17:45:45.510689 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.510568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpc85\" (UniqueName: \"kubernetes.io/projected/5f68b994-4a9c-49eb-a73e-8c9dab525227-kube-api-access-xpc85\") pod \"5f68b994-4a9c-49eb-a73e-8c9dab525227\" (UID: \"5f68b994-4a9c-49eb-a73e-8c9dab525227\") "
Jul 24 17:45:45.510689 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.510598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-plugins-config-volume\") pod \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\" (UID: \"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8\") "
Jul 24 17:45:45.511014 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.510989    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f68b994-4a9c-49eb-a73e-8c9dab525227-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f68b994-4a9c-49eb-a73e-8c9dab525227" (UID: "5f68b994-4a9c-49eb-a73e-8c9dab525227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:45:45.511114 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.511019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3718d43b-1db5-4b19-b4d8-ab1c415ba5d8" (UID: "3718d43b-1db5-4b19-b4d8-ab1c415ba5d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:45:45.512859 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.512840    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-kube-api-access-48hk8" (OuterVolumeSpecName: "kube-api-access-48hk8") pod "3718d43b-1db5-4b19-b4d8-ab1c415ba5d8" (UID: "3718d43b-1db5-4b19-b4d8-ab1c415ba5d8"). InnerVolumeSpecName "kube-api-access-48hk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:45:45.512957 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.512940    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f68b994-4a9c-49eb-a73e-8c9dab525227-kube-api-access-xpc85" (OuterVolumeSpecName: "kube-api-access-xpc85") pod "5f68b994-4a9c-49eb-a73e-8c9dab525227" (UID: "5f68b994-4a9c-49eb-a73e-8c9dab525227"). InnerVolumeSpecName "kube-api-access-xpc85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:45:45.612166 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.612139    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:45:45.612166 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.612164    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f68b994-4a9c-49eb-a73e-8c9dab525227-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:45:45.612303 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.612174    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48hk8\" (UniqueName: \"kubernetes.io/projected/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8-kube-api-access-48hk8\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:45:45.612303 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.612182    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpc85\" (UniqueName: \"kubernetes.io/projected/5f68b994-4a9c-49eb-a73e-8c9dab525227-kube-api-access-xpc85\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:45:45.901626 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.901597    2563 generic.go:356] "Generic (PLEG): container finished" podID="3718d43b-1db5-4b19-b4d8-ab1c415ba5d8" containerID="cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c" exitCode=0
Jul 24 17:45:45.901792 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.901642    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"
Jul 24 17:45:45.901792 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.901685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6" event={"ID":"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8","Type":"ContainerDied","Data":"cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c"}
Jul 24 17:45:45.901792 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.901745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6" event={"ID":"3718d43b-1db5-4b19-b4d8-ab1c415ba5d8","Type":"ContainerDied","Data":"7727b46fa1016044501da2ee4b2b0ffa68a53c5bf31854f3cc2d37f61a84b971"}
Jul 24 17:45:45.901792 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.901779    2563 scope.go:117] "RemoveContainer" containerID="cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c"
Jul 24 17:45:45.903073 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.903052    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f68b994-4a9c-49eb-a73e-8c9dab525227" containerID="d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585" exitCode=0
Jul 24 17:45:45.903173 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.903103    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"
Jul 24 17:45:45.903237 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.903136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx" event={"ID":"5f68b994-4a9c-49eb-a73e-8c9dab525227","Type":"ContainerDied","Data":"d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585"}
Jul 24 17:45:45.903370 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.903353    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx" event={"ID":"5f68b994-4a9c-49eb-a73e-8c9dab525227","Type":"ContainerDied","Data":"16f7fdf3afab3db6906de529dba7f3e3aad74e05b45cfaf541532a1b161dcf64"}
Jul 24 17:45:45.910359 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.910345    2563 scope.go:117] "RemoveContainer" containerID="cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c"
Jul 24 17:45:45.910603 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:45:45.910585    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c\": container with ID starting with cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c not found: ID does not exist" containerID="cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c"
Jul 24 17:45:45.910686 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.910612    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c"} err="failed to get container status \"cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c\": rpc error: code = NotFound desc = could not find container \"cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c\": container with ID starting with cb7a8d95e08067983953a3baec61c5400359a2b3d2255edfc423b6d2e382cf8c not found: ID does not exist"
Jul 24 17:45:45.910686 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.910636    2563 scope.go:117] "RemoveContainer" containerID="d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585"
Jul 24 17:45:45.918528 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.918513    2563 scope.go:117] "RemoveContainer" containerID="d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585"
Jul 24 17:45:45.918802 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:45:45.918781    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585\": container with ID starting with d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585 not found: ID does not exist" containerID="d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585"
Jul 24 17:45:45.918866 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.918810    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585"} err="failed to get container status \"d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585\": rpc error: code = NotFound desc = could not find container \"d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585\": container with ID starting with d9d1c63d3b90abb8ca0202b7424c48b99bbb9766fb1b79681aaedf74f6ce0585 not found: ID does not exist"
Jul 24 17:45:45.948611 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.948589    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"]
Jul 24 17:45:45.953822 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.953801    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac84f7-545b4dd48d-48brx"]
Jul 24 17:45:45.971049 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.971029    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"]
Jul 24 17:45:45.983913 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:45.983891    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac84f7-6f679bfb5d-fqkb6"]
Jul 24 17:45:46.688133 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:46.688100    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3718d43b-1db5-4b19-b4d8-ab1c415ba5d8" path="/var/lib/kubelet/pods/3718d43b-1db5-4b19-b4d8-ab1c415ba5d8/volumes"
Jul 24 17:45:46.688472 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:46.688428    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f68b994-4a9c-49eb-a73e-8c9dab525227" path="/var/lib/kubelet/pods/5f68b994-4a9c-49eb-a73e-8c9dab525227/volumes"
Jul 24 17:45:49.616341 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:49.616289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:50.066864 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:50.066833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:55.844800 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:55.844759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:56.013990 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:56.013955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:56.133591 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:56.133512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:45:56.371848 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:45:56.371817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:11.994375 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:11.994333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:12.021327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.021296    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"]
Jul 24 17:46:12.021559 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.021519    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr" podUID="f6c901ec-3d92-4d09-a538-fd51a74cb591" containerName="payload-processing" containerID="cri-o://518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a" gracePeriod=30
Jul 24 17:46:12.029398 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.029374    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"]
Jul 24 17:46:12.030130 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.030103    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c" podUID="4ade6270-04cc-401e-b168-61edb5d2e5a1" containerName="payload-pre-processing" containerID="cri-o://0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5" gracePeriod=30
Jul 24 17:46:12.095854 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.095825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:12.195016 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.194927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:12.268629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.268609    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:46:12.299641 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.299614    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:46:12.344964 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.344941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc59n\" (UniqueName: \"kubernetes.io/projected/4ade6270-04cc-401e-b168-61edb5d2e5a1-kube-api-access-hc59n\") pod \"4ade6270-04cc-401e-b168-61edb5d2e5a1\" (UID: \"4ade6270-04cc-401e-b168-61edb5d2e5a1\") "
Jul 24 17:46:12.345090 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.344989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtkjs\" (UniqueName: \"kubernetes.io/projected/f6c901ec-3d92-4d09-a538-fd51a74cb591-kube-api-access-jtkjs\") pod \"f6c901ec-3d92-4d09-a538-fd51a74cb591\" (UID: \"f6c901ec-3d92-4d09-a538-fd51a74cb591\") "
Jul 24 17:46:12.345090 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.345020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c901ec-3d92-4d09-a538-fd51a74cb591-plugins-config-volume\") pod \"f6c901ec-3d92-4d09-a538-fd51a74cb591\" (UID: \"f6c901ec-3d92-4d09-a538-fd51a74cb591\") "
Jul 24 17:46:12.345090 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.345072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade6270-04cc-401e-b168-61edb5d2e5a1-plugins-config-volume\") pod \"4ade6270-04cc-401e-b168-61edb5d2e5a1\" (UID: \"4ade6270-04cc-401e-b168-61edb5d2e5a1\") "
Jul 24 17:46:12.345475 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.345446    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c901ec-3d92-4d09-a538-fd51a74cb591-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6c901ec-3d92-4d09-a538-fd51a74cb591" (UID: "f6c901ec-3d92-4d09-a538-fd51a74cb591"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:46:12.345550 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.345448    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ade6270-04cc-401e-b168-61edb5d2e5a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ade6270-04cc-401e-b168-61edb5d2e5a1" (UID: "4ade6270-04cc-401e-b168-61edb5d2e5a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:46:12.347155 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.347134    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ade6270-04cc-401e-b168-61edb5d2e5a1-kube-api-access-hc59n" (OuterVolumeSpecName: "kube-api-access-hc59n") pod "4ade6270-04cc-401e-b168-61edb5d2e5a1" (UID: "4ade6270-04cc-401e-b168-61edb5d2e5a1"). InnerVolumeSpecName "kube-api-access-hc59n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:46:12.347321 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.347303    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c901ec-3d92-4d09-a538-fd51a74cb591-kube-api-access-jtkjs" (OuterVolumeSpecName: "kube-api-access-jtkjs") pod "f6c901ec-3d92-4d09-a538-fd51a74cb591" (UID: "f6c901ec-3d92-4d09-a538-fd51a74cb591"). InnerVolumeSpecName "kube-api-access-jtkjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:46:12.446104 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.446072    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc59n\" (UniqueName: \"kubernetes.io/projected/4ade6270-04cc-401e-b168-61edb5d2e5a1-kube-api-access-hc59n\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:46:12.446104 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.446105    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtkjs\" (UniqueName: \"kubernetes.io/projected/f6c901ec-3d92-4d09-a538-fd51a74cb591-kube-api-access-jtkjs\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:46:12.446104 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.446114    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c901ec-3d92-4d09-a538-fd51a74cb591-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:46:12.446268 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:12.446122    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade6270-04cc-401e-b168-61edb5d2e5a1-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:46:13.031179 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.031149    2563 generic.go:356] "Generic (PLEG): container finished" podID="f6c901ec-3d92-4d09-a538-fd51a74cb591" containerID="518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a" exitCode=0
Jul 24 17:46:13.031575 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.031217    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"
Jul 24 17:46:13.031575 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.031230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr" event={"ID":"f6c901ec-3d92-4d09-a538-fd51a74cb591","Type":"ContainerDied","Data":"518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a"}
Jul 24 17:46:13.031575 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.031286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr" event={"ID":"f6c901ec-3d92-4d09-a538-fd51a74cb591","Type":"ContainerDied","Data":"c035270e6672a40faed417ddd25e288b25b0dd789fe0efde29c86f0a5e365e06"}
Jul 24 17:46:13.031575 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.031319    2563 scope.go:117] "RemoveContainer" containerID="518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a"
Jul 24 17:46:13.033031 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.032880    2563 generic.go:356] "Generic (PLEG): container finished" podID="4ade6270-04cc-401e-b168-61edb5d2e5a1" containerID="0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5" exitCode=0
Jul 24 17:46:13.033031 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.032924    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"
Jul 24 17:46:13.033031 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.032960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c" event={"ID":"4ade6270-04cc-401e-b168-61edb5d2e5a1","Type":"ContainerDied","Data":"0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5"}
Jul 24 17:46:13.033031 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.032998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c" event={"ID":"4ade6270-04cc-401e-b168-61edb5d2e5a1","Type":"ContainerDied","Data":"69021cc89c9da37360773cc9b853f6aadedb964ece6bfd9e3d159a650280b7cb"}
Jul 24 17:46:13.051690 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.051670    2563 scope.go:117] "RemoveContainer" containerID="518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a"
Jul 24 17:46:13.051977 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:46:13.051955    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a\": container with ID starting with 518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a not found: ID does not exist" containerID="518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a"
Jul 24 17:46:13.052052 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.051984    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a"} err="failed to get container status \"518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a\": rpc error: code = NotFound desc = could not find container \"518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a\": container with ID starting with 518cfa3a1ce331c625e5f2f52f412aa1281dbf39d32ba586680256f829990b5a not found: ID does not exist"
Jul 24 17:46:13.052052 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.052009    2563 scope.go:117] "RemoveContainer" containerID="0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5"
Jul 24 17:46:13.061260 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.061232    2563 scope.go:117] "RemoveContainer" containerID="0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5"
Jul 24 17:46:13.061381 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.061286    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"]
Jul 24 17:46:13.061533 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:46:13.061516    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5\": container with ID starting with 0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5 not found: ID does not exist" containerID="0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5"
Jul 24 17:46:13.061588 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.061539    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5"} err="failed to get container status \"0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5\": rpc error: code = NotFound desc = could not find container \"0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5\": container with ID starting with 0eabed6a6de1e64926b92cfe8c11c48d50e2bb67e7a95c09645dd2fb9eb692f5 not found: ID does not exist"
Jul 24 17:46:13.066567 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.066545    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-94ab01-79b45ccbcb-mx4cr"]
Jul 24 17:46:13.076316 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.076293    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"]
Jul 24 17:46:13.082747 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:13.082725    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-94ab01-7cccc69dd4-lh22c"]
Jul 24 17:46:14.688217 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:14.688177    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ade6270-04cc-401e-b168-61edb5d2e5a1" path="/var/lib/kubelet/pods/4ade6270-04cc-401e-b168-61edb5d2e5a1/volumes"
Jul 24 17:46:14.688638 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:14.688536    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c901ec-3d92-4d09-a538-fd51a74cb591" path="/var/lib/kubelet/pods/f6c901ec-3d92-4d09-a538-fd51a74cb591/volumes"
Jul 24 17:46:16.582265 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:16.582236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:16.983066 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:16.983037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:23.041771 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:23.041735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:23.141694 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:23.141632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:23.332103 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:23.332019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:23.570223 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:23.570192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:23.931929 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:23.931890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:24.252073 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:24.252042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:24.596695 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:24.596594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:27.200152 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:27.200103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:27.300872 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:27.300840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:27.511051 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:27.511013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:27.623938 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:27.623906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:27.996301 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:27.996266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:28.097352 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.097317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:28.123862 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.123827    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"]
Jul 24 17:46:28.124210 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124197    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ade6270-04cc-401e-b168-61edb5d2e5a1" containerName="payload-pre-processing"
Jul 24 17:46:28.124254 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124211    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ade6270-04cc-401e-b168-61edb5d2e5a1" containerName="payload-pre-processing"
Jul 24 17:46:28.124254 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124224    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3718d43b-1db5-4b19-b4d8-ab1c415ba5d8" containerName="payload-processing"
Jul 24 17:46:28.124254 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124231    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3718d43b-1db5-4b19-b4d8-ab1c415ba5d8" containerName="payload-processing"
Jul 24 17:46:28.124254 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124245    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c901ec-3d92-4d09-a538-fd51a74cb591" containerName="payload-processing"
Jul 24 17:46:28.124254 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124251    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c901ec-3d92-4d09-a538-fd51a74cb591" containerName="payload-processing"
Jul 24 17:46:28.124440 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124258    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f68b994-4a9c-49eb-a73e-8c9dab525227" containerName="payload-pre-processing"
Jul 24 17:46:28.124440 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124263    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f68b994-4a9c-49eb-a73e-8c9dab525227" containerName="payload-pre-processing"
Jul 24 17:46:28.124440 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124312    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ade6270-04cc-401e-b168-61edb5d2e5a1" containerName="payload-pre-processing"
Jul 24 17:46:28.124440 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124319    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3718d43b-1db5-4b19-b4d8-ab1c415ba5d8" containerName="payload-processing"
Jul 24 17:46:28.124440 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124328    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f68b994-4a9c-49eb-a73e-8c9dab525227" containerName="payload-pre-processing"
Jul 24 17:46:28.124440 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.124334    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c901ec-3d92-4d09-a538-fd51a74cb591" containerName="payload-processing"
Jul 24 17:46:28.127382 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.127363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:28.129930 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.129906    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"]
Jul 24 17:46:28.130480 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.130460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-13c2c7-dockercfg-pmw8b\""
Jul 24 17:46:28.131036 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.131019    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-13c2c7\""
Jul 24 17:46:28.133286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.133268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:28.148109 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.148083    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"]
Jul 24 17:46:28.152102 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.152082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"]
Jul 24 17:46:28.186268 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.186238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl9lx\" (UniqueName: \"kubernetes.io/projected/56cf44cd-e826-488b-83ee-11abfdb1a613-kube-api-access-jl9lx\") pod \"payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq\" (UID: \"56cf44cd-e826-488b-83ee-11abfdb1a613\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:28.186445 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.186278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cf44cd-e826-488b-83ee-11abfdb1a613-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq\" (UID: \"56cf44cd-e826-488b-83ee-11abfdb1a613\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:28.186445 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.186300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz\" (UID: \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:28.186445 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.186365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdm7t\" (UniqueName: \"kubernetes.io/projected/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-kube-api-access-cdm7t\") pod \"payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz\" (UID: \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:28.287326 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.287236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl9lx\" (UniqueName: \"kubernetes.io/projected/56cf44cd-e826-488b-83ee-11abfdb1a613-kube-api-access-jl9lx\") pod \"payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq\" (UID: \"56cf44cd-e826-488b-83ee-11abfdb1a613\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:28.287326 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.287299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cf44cd-e826-488b-83ee-11abfdb1a613-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq\" (UID: \"56cf44cd-e826-488b-83ee-11abfdb1a613\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:28.287757 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.287336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz\" (UID: \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:28.287757 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.287364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdm7t\" (UniqueName: \"kubernetes.io/projected/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-kube-api-access-cdm7t\") pod \"payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz\" (UID: \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:28.288053 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.288033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cf44cd-e826-488b-83ee-11abfdb1a613-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq\" (UID: \"56cf44cd-e826-488b-83ee-11abfdb1a613\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:28.288095 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.288079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz\" (UID: \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:28.295746 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.295720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl9lx\" (UniqueName: \"kubernetes.io/projected/56cf44cd-e826-488b-83ee-11abfdb1a613-kube-api-access-jl9lx\") pod \"payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq\" (UID: \"56cf44cd-e826-488b-83ee-11abfdb1a613\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:28.295866 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.295850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdm7t\" (UniqueName: \"kubernetes.io/projected/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-kube-api-access-cdm7t\") pod \"payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz\" (UID: \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:28.437259 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.437218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:28.445471 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.445448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:28.599723 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.599691    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"]
Jul 24 17:46:28.603099 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:46:28.603060    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41f9704_6dad_4b1a_903a_a4c8f3cfadfe.slice/crio-3a6e38e3c9131f52799223c656f2eeb13bcbae4766288d87cb056d67412b3340 WatchSource:0}: Error finding container 3a6e38e3c9131f52799223c656f2eeb13bcbae4766288d87cb056d67412b3340: Status 404 returned error can't find the container with id 3a6e38e3c9131f52799223c656f2eeb13bcbae4766288d87cb056d67412b3340
Jul 24 17:46:28.835091 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:28.834864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"]
Jul 24 17:46:28.837814 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:46:28.837774    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cf44cd_e826_488b_83ee_11abfdb1a613.slice/crio-d7e5c4a780c0b366fbbf1ac860d9e12b675e449f3077b51d6c1ad5e1d3a21907 WatchSource:0}: Error finding container d7e5c4a780c0b366fbbf1ac860d9e12b675e449f3077b51d6c1ad5e1d3a21907: Status 404 returned error can't find the container with id d7e5c4a780c0b366fbbf1ac860d9e12b675e449f3077b51d6c1ad5e1d3a21907
Jul 24 17:46:29.109576 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:29.109487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq" event={"ID":"56cf44cd-e826-488b-83ee-11abfdb1a613","Type":"ContainerStarted","Data":"0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2"}
Jul 24 17:46:29.109576 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:29.109555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:29.109827 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:29.109578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq" event={"ID":"56cf44cd-e826-488b-83ee-11abfdb1a613","Type":"ContainerStarted","Data":"d7e5c4a780c0b366fbbf1ac860d9e12b675e449f3077b51d6c1ad5e1d3a21907"}
Jul 24 17:46:29.111212 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:29.111186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz" event={"ID":"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe","Type":"ContainerStarted","Data":"d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a"}
Jul 24 17:46:29.111418 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:29.111381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz" event={"ID":"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe","Type":"ContainerStarted","Data":"3a6e38e3c9131f52799223c656f2eeb13bcbae4766288d87cb056d67412b3340"}
Jul 24 17:46:29.111533 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:29.111423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:29.128634 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:29.128589    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq" podStartSLOduration=1.128573861 podStartE2EDuration="1.128573861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:29.126970591 +0000 UTC m=+2592.958223279" watchObservedRunningTime="2026-07-24 17:46:29.128573861 +0000 UTC m=+2592.959826611"
Jul 24 17:46:29.145411 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:29.145367    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz" podStartSLOduration=1.145356518 podStartE2EDuration="1.145356518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:29.143086027 +0000 UTC m=+2592.974338713" watchObservedRunningTime="2026-07-24 17:46:29.145356518 +0000 UTC m=+2592.976609205"
Jul 24 17:46:31.730797 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:31.730739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:32.134142 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:32.134001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:32.423126 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:32.423003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:32.572102 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:32.572071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:35.275855 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:35.275807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:35.371492 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:35.371455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:35.450663 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:35.450624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:35.591767 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:35.591641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:35.957395 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:35.957277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:36.056964 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.056905    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"]
Jul 24 17:46:36.058191 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.058141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:36.060291 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.060276    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:36.062568 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.062548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-b9a122\""
Jul 24 17:46:36.071280 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.071256    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"]
Jul 24 17:46:36.101836 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.101806    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"]
Jul 24 17:46:36.105366 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.105346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:36.107878 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.107860    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-b9a122-dockercfg-4ghph\""
Jul 24 17:46:36.118937 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.118916    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"]
Jul 24 17:46:36.159234 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.159203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg\" (UID: \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:36.159380 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.159254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvknq\" (UniqueName: \"kubernetes.io/projected/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-kube-api-access-gvknq\") pod \"payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg\" (UID: \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:36.260054 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.260019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg\" (UID: \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:36.260229 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.260060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpr9c\" (UniqueName: \"kubernetes.io/projected/29a79bbb-6304-4243-a20a-b862a7f2bb26-kube-api-access-vpr9c\") pod \"payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm\" (UID: \"29a79bbb-6304-4243-a20a-b862a7f2bb26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:36.260229 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.260101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a79bbb-6304-4243-a20a-b862a7f2bb26-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm\" (UID: \"29a79bbb-6304-4243-a20a-b862a7f2bb26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:36.260229 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.260131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvknq\" (UniqueName: \"kubernetes.io/projected/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-kube-api-access-gvknq\") pod \"payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg\" (UID: \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:36.260707 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.260687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg\" (UID: \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:36.268941 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.268921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvknq\" (UniqueName: \"kubernetes.io/projected/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-kube-api-access-gvknq\") pod \"payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg\" (UID: \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:36.360760 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.360733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a79bbb-6304-4243-a20a-b862a7f2bb26-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm\" (UID: \"29a79bbb-6304-4243-a20a-b862a7f2bb26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:36.361177 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.360833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpr9c\" (UniqueName: \"kubernetes.io/projected/29a79bbb-6304-4243-a20a-b862a7f2bb26-kube-api-access-vpr9c\") pod \"payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm\" (UID: \"29a79bbb-6304-4243-a20a-b862a7f2bb26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:36.361513 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.361489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a79bbb-6304-4243-a20a-b862a7f2bb26-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm\" (UID: \"29a79bbb-6304-4243-a20a-b862a7f2bb26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:36.370334 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.370310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpr9c\" (UniqueName: \"kubernetes.io/projected/29a79bbb-6304-4243-a20a-b862a7f2bb26-kube-api-access-vpr9c\") pod \"payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm\" (UID: \"29a79bbb-6304-4243-a20a-b862a7f2bb26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:36.372094 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.372077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:36.415299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.415268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:36.523107 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.523083    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"]
Jul 24 17:46:36.525500 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:46:36.525458    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a95ef01_847e_4ad4_ae3e_89a7c9a1cf00.slice/crio-d6469060c4da534e4fa5dc323d1fbbbe31cc0be6df524ee373f66545da22f9d0 WatchSource:0}: Error finding container d6469060c4da534e4fa5dc323d1fbbbe31cc0be6df524ee373f66545da22f9d0: Status 404 returned error can't find the container with id d6469060c4da534e4fa5dc323d1fbbbe31cc0be6df524ee373f66545da22f9d0
Jul 24 17:46:36.578026 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:36.577940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"]
Jul 24 17:46:36.580248 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:46:36.580221    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a79bbb_6304_4243_a20a_b862a7f2bb26.slice/crio-b8f79de4466649881b47d786f18675d2591acd58b3d5cf8df89d07a236dd6a8a WatchSource:0}: Error finding container b8f79de4466649881b47d786f18675d2591acd58b3d5cf8df89d07a236dd6a8a: Status 404 returned error can't find the container with id b8f79de4466649881b47d786f18675d2591acd58b3d5cf8df89d07a236dd6a8a
Jul 24 17:46:37.151402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:37.151317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg" event={"ID":"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00","Type":"ContainerStarted","Data":"31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56"}
Jul 24 17:46:37.151402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:37.151374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg" event={"ID":"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00","Type":"ContainerStarted","Data":"d6469060c4da534e4fa5dc323d1fbbbe31cc0be6df524ee373f66545da22f9d0"}
Jul 24 17:46:37.151634 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:37.151402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:37.152879 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:37.152855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm" event={"ID":"29a79bbb-6304-4243-a20a-b862a7f2bb26","Type":"ContainerStarted","Data":"818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478"}
Jul 24 17:46:37.152989 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:37.152896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm" event={"ID":"29a79bbb-6304-4243-a20a-b862a7f2bb26","Type":"ContainerStarted","Data":"b8f79de4466649881b47d786f18675d2591acd58b3d5cf8df89d07a236dd6a8a"}
Jul 24 17:46:37.152989 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:37.152918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:37.169312 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:37.169261    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg" podStartSLOduration=1.169247577 podStartE2EDuration="1.169247577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:37.167477506 +0000 UTC m=+2600.998730193" watchObservedRunningTime="2026-07-24 17:46:37.169247577 +0000 UTC m=+2601.000500302"
Jul 24 17:46:37.185450 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:37.185409    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm" podStartSLOduration=1.1853993 podStartE2EDuration="1.1853993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:37.18386661 +0000 UTC m=+2601.015119297" watchObservedRunningTime="2026-07-24 17:46:37.1853993 +0000 UTC m=+2601.016651986"
Jul 24 17:46:39.731728 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:39.731693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:39.831443 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:39.831411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:40.118472 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:40.118394    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:46:40.118613 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:40.118477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:46:48.159263 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:48.159232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:46:48.159709 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:48.159315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:46:52.947701 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:52.947665    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"]
Jul 24 17:46:52.951276 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:52.951256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:52.954624 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:52.954603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-13c2c7\"/\"openshift-service-ca.crt\""
Jul 24 17:46:52.954778 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:52.954607    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-13c2c7\"/\"kube-root-ca.crt\""
Jul 24 17:46:52.954778 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:52.954608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-13c2c7\"/\"default-dockercfg-xplst\""
Jul 24 17:46:52.954778 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:52.954690    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-13c2c7\"/\"auth-test-model-13c2c7-kserve-self-signed-certs\""
Jul 24 17:46:52.962224 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:52.962204    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"]
Jul 24 17:46:53.008553 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.008525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdtrk\" (UniqueName: \"kubernetes.io/projected/c8a2df38-bec4-4146-9d39-dd1f72c2d693-kube-api-access-mdtrk\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.008711 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.008562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tmp-dir\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.008711 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.008598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-home\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.008711 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.008625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-model-cache\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.008711 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.008676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tls-certs\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.008876 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.008759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-dshm\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.037485 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.037459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:46:53.109692 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.109670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tls-certs\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.109806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.109713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-dshm\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.109806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.109762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdtrk\" (UniqueName: \"kubernetes.io/projected/c8a2df38-bec4-4146-9d39-dd1f72c2d693-kube-api-access-mdtrk\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.109806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.109783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tmp-dir\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.109806 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.109803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-home\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.110015 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.109819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-model-cache\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.110241 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.110220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tmp-dir\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.110306 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.110261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-home\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.110345 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.110305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-model-cache\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.112154 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.112136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-dshm\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.112283 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.112267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tls-certs\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.117859 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.117840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdtrk\" (UniqueName: \"kubernetes.io/projected/c8a2df38-bec4-4146-9d39-dd1f72c2d693-kube-api-access-mdtrk\") pod \"auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") " pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.261409 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.261382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:46:53.395882 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:53.395858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"]
Jul 24 17:46:53.397863 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:46:53.397838    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a2df38_bec4_4146_9d39_dd1f72c2d693.slice/crio-5526a4d8fb48897a6a1a48fe3c6217a8c831d0db0b46075ce91000090c27d71c WatchSource:0}: Error finding container 5526a4d8fb48897a6a1a48fe3c6217a8c831d0db0b46075ce91000090c27d71c: Status 404 returned error can't find the container with id 5526a4d8fb48897a6a1a48fe3c6217a8c831d0db0b46075ce91000090c27d71c
Jul 24 17:46:54.237080 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:54.237044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8" event={"ID":"c8a2df38-bec4-4146-9d39-dd1f72c2d693","Type":"ContainerStarted","Data":"4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca"}
Jul 24 17:46:54.237080 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:54.237085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8" event={"ID":"c8a2df38-bec4-4146-9d39-dd1f72c2d693","Type":"ContainerStarted","Data":"5526a4d8fb48897a6a1a48fe3c6217a8c831d0db0b46075ce91000090c27d71c"}
Jul 24 17:46:54.254309 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:46:54.254266    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8" podStartSLOduration=2.254254401 podStartE2EDuration="2.254254401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:54.253310279 +0000 UTC m=+2618.084562967" watchObservedRunningTime="2026-07-24 17:46:54.254254401 +0000 UTC m=+2618.085507088"
Jul 24 17:47:03.261795 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:47:03.261757    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:47:03.264379 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:47:03.261819    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:47:03.274317 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:47:03.274299    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:47:03.290135 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:47:03.290107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:47:10.158566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:47:10.158536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:47:41.250828 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:47:41.250780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:10.200066 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:48:10.200027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:10.300521 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:48:10.300494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:48:16.789110 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:48:16.789085    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:48:16.795306 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:48:16.795287    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:49:11.934278 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:11.934239    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"]
Jul 24 17:49:11.954813 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:11.954771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:12.844267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:12.844231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:26.460233 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.460196    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"]
Jul 24 17:49:26.463734 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.463718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.466214 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.466196    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-13c2c7\"/\"sub-test-model-13c2c7-kserve-self-signed-certs\""
Jul 24 17:49:26.476079 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.476059    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"]
Jul 24 17:49:26.559070 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.559023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:49:26.573165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.573142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-tmp-dir\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.573235 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.573199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-model-cache\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.573272 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.573243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-home\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.573307 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.573274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-dshm\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.573307 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.573296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzj5q\" (UniqueName: \"kubernetes.io/projected/f24ab447-b8fd-4a3e-8542-5154928e0e13-kube-api-access-xzj5q\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.573377 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.573341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ab447-b8fd-4a3e-8542-5154928e0e13-tls-certs\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.674518 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.674486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-dshm\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.674518 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.674518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzj5q\" (UniqueName: \"kubernetes.io/projected/f24ab447-b8fd-4a3e-8542-5154928e0e13-kube-api-access-xzj5q\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.674749 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.674555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ab447-b8fd-4a3e-8542-5154928e0e13-tls-certs\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.674749 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.674597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-tmp-dir\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.674749 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.674697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-model-cache\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.674749 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.674726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-home\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.675165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.675130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-model-cache\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.675165 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.675155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-tmp-dir\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.683406 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.675540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-home\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.683406 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.683218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-dshm\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.683406 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.683347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ab447-b8fd-4a3e-8542-5154928e0e13-tls-certs\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.684073 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.684049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzj5q\" (UniqueName: \"kubernetes.io/projected/f24ab447-b8fd-4a3e-8542-5154928e0e13-kube-api-access-xzj5q\") pod \"sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") " pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.772479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.772451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:26.904137 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.904111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"]
Jul 24 17:49:26.906482 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:49:26.906453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf24ab447_b8fd_4a3e_8542_5154928e0e13.slice/crio-8080a2dc91f8c4a69f264bb7a7911f14891be2e6600a0e008031629601b22018 WatchSource:0}: Error finding container 8080a2dc91f8c4a69f264bb7a7911f14891be2e6600a0e008031629601b22018: Status 404 returned error can't find the container with id 8080a2dc91f8c4a69f264bb7a7911f14891be2e6600a0e008031629601b22018
Jul 24 17:49:26.953330 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.953302    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8" podUID="c8a2df38-bec4-4146-9d39-dd1f72c2d693" containerName="main" containerID="cri-o://4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca" gracePeriod=45
Jul 24 17:49:26.962639 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:26.962614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6" event={"ID":"f24ab447-b8fd-4a3e-8542-5154928e0e13","Type":"ContainerStarted","Data":"8080a2dc91f8c4a69f264bb7a7911f14891be2e6600a0e008031629601b22018"}
Jul 24 17:49:27.183711 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.183689    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:49:27.280357 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280331    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tmp-dir\") pod \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") "
Jul 24 17:49:27.280551 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-dshm\") pod \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") "
Jul 24 17:49:27.280551 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280422    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-model-cache\") pod \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") "
Jul 24 17:49:27.280551 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280483    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-home\") pod \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") "
Jul 24 17:49:27.280551 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tls-certs\") pod \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") "
Jul 24 17:49:27.280786 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdtrk\" (UniqueName: \"kubernetes.io/projected/c8a2df38-bec4-4146-9d39-dd1f72c2d693-kube-api-access-mdtrk\") pod \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\" (UID: \"c8a2df38-bec4-4146-9d39-dd1f72c2d693\") "
Jul 24 17:49:27.280786 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8a2df38-bec4-4146-9d39-dd1f72c2d693" (UID: "c8a2df38-bec4-4146-9d39-dd1f72c2d693"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:49:27.280786 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280731    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8a2df38-bec4-4146-9d39-dd1f72c2d693" (UID: "c8a2df38-bec4-4146-9d39-dd1f72c2d693"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:49:27.280786 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-home" (OuterVolumeSpecName: "home") pod "c8a2df38-bec4-4146-9d39-dd1f72c2d693" (UID: "c8a2df38-bec4-4146-9d39-dd1f72c2d693"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:49:27.280969 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280954    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tmp-dir\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:49:27.281039 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280973    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-model-cache\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:49:27.281039 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.280987    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-home\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:49:27.282807 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.282779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-dshm" (OuterVolumeSpecName: "dshm") pod "c8a2df38-bec4-4146-9d39-dd1f72c2d693" (UID: "c8a2df38-bec4-4146-9d39-dd1f72c2d693"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:49:27.282910 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.282822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a2df38-bec4-4146-9d39-dd1f72c2d693-kube-api-access-mdtrk" (OuterVolumeSpecName: "kube-api-access-mdtrk") pod "c8a2df38-bec4-4146-9d39-dd1f72c2d693" (UID: "c8a2df38-bec4-4146-9d39-dd1f72c2d693"). InnerVolumeSpecName "kube-api-access-mdtrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:49:27.282910 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.282844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8a2df38-bec4-4146-9d39-dd1f72c2d693" (UID: "c8a2df38-bec4-4146-9d39-dd1f72c2d693"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:49:27.381669 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.381581    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2df38-bec4-4146-9d39-dd1f72c2d693-dshm\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:49:27.381669 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.381607    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2df38-bec4-4146-9d39-dd1f72c2d693-tls-certs\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:49:27.381669 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.381617    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdtrk\" (UniqueName: \"kubernetes.io/projected/c8a2df38-bec4-4146-9d39-dd1f72c2d693-kube-api-access-mdtrk\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:49:27.968955 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.968921    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8a2df38-bec4-4146-9d39-dd1f72c2d693" containerID="4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca" exitCode=0
Jul 24 17:49:27.969399 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.968961    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"
Jul 24 17:49:27.969399 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.969006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8" event={"ID":"c8a2df38-bec4-4146-9d39-dd1f72c2d693","Type":"ContainerDied","Data":"4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca"}
Jul 24 17:49:27.969399 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.969052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8" event={"ID":"c8a2df38-bec4-4146-9d39-dd1f72c2d693","Type":"ContainerDied","Data":"5526a4d8fb48897a6a1a48fe3c6217a8c831d0db0b46075ce91000090c27d71c"}
Jul 24 17:49:27.969399 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.969077    2563 scope.go:117] "RemoveContainer" containerID="4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca"
Jul 24 17:49:27.970627 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.970603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6" event={"ID":"f24ab447-b8fd-4a3e-8542-5154928e0e13","Type":"ContainerStarted","Data":"d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37"}
Jul 24 17:49:27.977867 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.977852    2563 scope.go:117] "RemoveContainer" containerID="4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca"
Jul 24 17:49:27.978152 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:49:27.978137    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca\": container with ID starting with 4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca not found: ID does not exist" containerID="4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca"
Jul 24 17:49:27.978203 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.978160    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca"} err="failed to get container status \"4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca\": rpc error: code = NotFound desc = could not find container \"4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca\": container with ID starting with 4ec5f936da119a0c917f08cf5f256b9559887d9c957b29951024e47edbf0daca not found: ID does not exist"
Jul 24 17:49:27.989581 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:27.989545    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6" podStartSLOduration=1.989533735 podStartE2EDuration="1.989533735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:27.987403897 +0000 UTC m=+2771.818656583" watchObservedRunningTime="2026-07-24 17:49:27.989533735 +0000 UTC m=+2771.820786420"
Jul 24 17:49:28.002736 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:28.002717    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"]
Jul 24 17:49:28.006688 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:28.006669    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/auth-test-model-13c2c7-kserve-69dd9fbcf5-k8rr8"]
Jul 24 17:49:28.689123 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:28.689091    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a2df38-bec4-4146-9d39-dd1f72c2d693" path="/var/lib/kubelet/pods/c8a2df38-bec4-4146-9d39-dd1f72c2d693/volumes"
Jul 24 17:49:36.773498 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:36.773465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:36.774025 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:36.773512    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:36.785696 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:36.785672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:37.024992 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:37.024873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:49:58.284883 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:49:58.284853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:04.687662 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:04.687620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:42.637723 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:42.637691    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"]
Jul 24 17:50:42.658113 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:42.658070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:43.688065 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:43.688033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:55.552079 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.552037    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"]
Jul 24 17:50:55.552685 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.552622    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a2df38-bec4-4146-9d39-dd1f72c2d693" containerName="main"
Jul 24 17:50:55.552685 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.552642    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a2df38-bec4-4146-9d39-dd1f72c2d693" containerName="main"
Jul 24 17:50:55.552829 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.552769    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a2df38-bec4-4146-9d39-dd1f72c2d693" containerName="main"
Jul 24 17:50:55.555945 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.555924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.558199 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.558180    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-13c2c7\"/\"rate-test-model-13c2c7-kserve-self-signed-certs\""
Jul 24 17:50:55.565974 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.565953    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"]
Jul 24 17:50:55.620847 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.620822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-dshm\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.621021 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.620859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tls-certs\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.621021 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.620880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4hr\" (UniqueName: \"kubernetes.io/projected/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-kube-api-access-gk4hr\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.621021 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.620977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-model-cache\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.621187 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.621024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tmp-dir\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.621187 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.621125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-home\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.721827 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.721798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-home\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.721977 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.721843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-dshm\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.721977 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.721873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tls-certs\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.721977 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.721900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4hr\" (UniqueName: \"kubernetes.io/projected/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-kube-api-access-gk4hr\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.721977 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.721963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-model-cache\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.722206 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.721989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tmp-dir\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.722315 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.722258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-home\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.722395 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.722333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tmp-dir\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.722569 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.722537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-model-cache\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.724686 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.724636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-dshm\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.724939 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.724921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tls-certs\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.730273 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.730254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4hr\" (UniqueName: \"kubernetes.io/projected/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-kube-api-access-gk4hr\") pod \"rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") " pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:55.755302 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.755271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:50:55.865346 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:55.865270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:50:56.005880 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:56.005852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"]
Jul 24 17:50:56.008576 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:50:56.008552    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fae2701_ad4f_48f4_87c1_3e7085bdbd1e.slice/crio-f7d8f4e26517e5f7e0090f0d1d4267e650c317114c7336b41f92bc11d936a002 WatchSource:0}: Error finding container f7d8f4e26517e5f7e0090f0d1d4267e650c317114c7336b41f92bc11d936a002: Status 404 returned error can't find the container with id f7d8f4e26517e5f7e0090f0d1d4267e650c317114c7336b41f92bc11d936a002
Jul 24 17:50:56.398237 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:56.398196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv" event={"ID":"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e","Type":"ContainerStarted","Data":"4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927"}
Jul 24 17:50:56.398237 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:56.398242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv" event={"ID":"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e","Type":"ContainerStarted","Data":"f7d8f4e26517e5f7e0090f0d1d4267e650c317114c7336b41f92bc11d936a002"}
Jul 24 17:50:56.416578 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:56.416530    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv" podStartSLOduration=1.416515089 podStartE2EDuration="1.416515089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:56.414367948 +0000 UTC m=+2860.245620638" watchObservedRunningTime="2026-07-24 17:50:56.416515089 +0000 UTC m=+2860.247767817"
Jul 24 17:50:57.657020 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:57.656982    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6" podUID="f24ab447-b8fd-4a3e-8542-5154928e0e13" containerName="main" containerID="cri-o://d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37" gracePeriod=45
Jul 24 17:50:57.923705 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:57.923677    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:50:58.045090 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzj5q\" (UniqueName: \"kubernetes.io/projected/f24ab447-b8fd-4a3e-8542-5154928e0e13-kube-api-access-xzj5q\") pod \"f24ab447-b8fd-4a3e-8542-5154928e0e13\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") "
Jul 24 17:50:58.045273 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045113    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-tmp-dir\") pod \"f24ab447-b8fd-4a3e-8542-5154928e0e13\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") "
Jul 24 17:50:58.045273 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045134    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ab447-b8fd-4a3e-8542-5154928e0e13-tls-certs\") pod \"f24ab447-b8fd-4a3e-8542-5154928e0e13\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") "
Jul 24 17:50:58.045273 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-model-cache\") pod \"f24ab447-b8fd-4a3e-8542-5154928e0e13\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") "
Jul 24 17:50:58.045273 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-home\") pod \"f24ab447-b8fd-4a3e-8542-5154928e0e13\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") "
Jul 24 17:50:58.045485 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045294    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-dshm\") pod \"f24ab447-b8fd-4a3e-8542-5154928e0e13\" (UID: \"f24ab447-b8fd-4a3e-8542-5154928e0e13\") "
Jul 24 17:50:58.045485 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045339    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f24ab447-b8fd-4a3e-8542-5154928e0e13" (UID: "f24ab447-b8fd-4a3e-8542-5154928e0e13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:50:58.045584 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045490    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-model-cache" (OuterVolumeSpecName: "model-cache") pod "f24ab447-b8fd-4a3e-8542-5154928e0e13" (UID: "f24ab447-b8fd-4a3e-8542-5154928e0e13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:50:58.045584 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045509    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-home" (OuterVolumeSpecName: "home") pod "f24ab447-b8fd-4a3e-8542-5154928e0e13" (UID: "f24ab447-b8fd-4a3e-8542-5154928e0e13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:50:58.045684 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045606    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-tmp-dir\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:50:58.045684 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045618    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-model-cache\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:50:58.045684 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.045627    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-home\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:50:58.047402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.047385    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-dshm" (OuterVolumeSpecName: "dshm") pod "f24ab447-b8fd-4a3e-8542-5154928e0e13" (UID: "f24ab447-b8fd-4a3e-8542-5154928e0e13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:50:58.047470 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.047443    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24ab447-b8fd-4a3e-8542-5154928e0e13-kube-api-access-xzj5q" (OuterVolumeSpecName: "kube-api-access-xzj5q") pod "f24ab447-b8fd-4a3e-8542-5154928e0e13" (UID: "f24ab447-b8fd-4a3e-8542-5154928e0e13"). InnerVolumeSpecName "kube-api-access-xzj5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:50:58.047509 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.047466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f24ab447-b8fd-4a3e-8542-5154928e0e13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f24ab447-b8fd-4a3e-8542-5154928e0e13" (UID: "f24ab447-b8fd-4a3e-8542-5154928e0e13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:50:58.147149 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.147103    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24ab447-b8fd-4a3e-8542-5154928e0e13-dshm\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:50:58.147149 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.147145    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzj5q\" (UniqueName: \"kubernetes.io/projected/f24ab447-b8fd-4a3e-8542-5154928e0e13-kube-api-access-xzj5q\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:50:58.147149 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.147155    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24ab447-b8fd-4a3e-8542-5154928e0e13-tls-certs\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:50:58.409185 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.409151    2563 generic.go:356] "Generic (PLEG): container finished" podID="f24ab447-b8fd-4a3e-8542-5154928e0e13" containerID="d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37" exitCode=0
Jul 24 17:50:58.409349 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.409198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"
Jul 24 17:50:58.409349 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.409200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6" event={"ID":"f24ab447-b8fd-4a3e-8542-5154928e0e13","Type":"ContainerDied","Data":"d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37"}
Jul 24 17:50:58.409349 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.409313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6" event={"ID":"f24ab447-b8fd-4a3e-8542-5154928e0e13","Type":"ContainerDied","Data":"8080a2dc91f8c4a69f264bb7a7911f14891be2e6600a0e008031629601b22018"}
Jul 24 17:50:58.409349 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.409338    2563 scope.go:117] "RemoveContainer" containerID="d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37"
Jul 24 17:50:58.418429 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.418412    2563 scope.go:117] "RemoveContainer" containerID="d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37"
Jul 24 17:50:58.418752 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:50:58.418733    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37\": container with ID starting with d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37 not found: ID does not exist" containerID="d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37"
Jul 24 17:50:58.418820 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.418761    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37"} err="failed to get container status \"d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37\": rpc error: code = NotFound desc = could not find container \"d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37\": container with ID starting with d5aed6a732add7e98396a822f250a73b41a82ab41aab60df2209b27ea9ea4c37 not found: ID does not exist"
Jul 24 17:50:58.430262 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.430239    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"]
Jul 24 17:50:58.434312 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.434291    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/sub-test-model-13c2c7-kserve-7fb9ff8889-d6mg6"]
Jul 24 17:50:58.688699 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:50:58.688588    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f24ab447-b8fd-4a3e-8542-5154928e0e13" path="/var/lib/kubelet/pods/f24ab447-b8fd-4a3e-8542-5154928e0e13/volumes"
Jul 24 17:51:05.865974 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:51:05.865936    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:51:05.866394 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:51:05.866000    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:51:05.878371 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:51:05.878340    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:51:06.457681 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:51:06.457623    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:51:17.777339 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:51:17.777302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:51:44.103804 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:51:44.103768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:02.164154 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:02.164116    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"]
Jul 24 17:52:02.179877 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:02.179847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:03.162695 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:03.162666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:17.183681 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.183610    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv" podUID="9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" containerName="main" containerID="cri-o://4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927" gracePeriod=45
Jul 24 17:52:17.437244 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.437184    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:52:17.493402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.493363    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-model-cache\") pod \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") "
Jul 24 17:52:17.493554 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.493439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk4hr\" (UniqueName: \"kubernetes.io/projected/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-kube-api-access-gk4hr\") pod \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") "
Jul 24 17:52:17.493554 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.493513    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-home\") pod \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") "
Jul 24 17:52:17.493554 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.493548    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tmp-dir\") pod \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") "
Jul 24 17:52:17.493785 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.493599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-dshm\") pod \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") "
Jul 24 17:52:17.493785 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.493672    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tls-certs\") pod \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\" (UID: \"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e\") "
Jul 24 17:52:17.493785 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.493705    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" (UID: "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:52:17.493949 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.493804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-home" (OuterVolumeSpecName: "home") pod "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" (UID: "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:52:17.493949 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.493855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" (UID: "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:52:17.494075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.494020    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-home\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:52:17.494075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.494064    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tmp-dir\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:52:17.494159 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.494079    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-model-cache\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:52:17.495867 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.495841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-dshm" (OuterVolumeSpecName: "dshm") pod "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" (UID: "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:52:17.495996 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.495903    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" (UID: "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:52:17.496155 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.496125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-kube-api-access-gk4hr" (OuterVolumeSpecName: "kube-api-access-gk4hr") pod "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" (UID: "9fae2701-ad4f-48f4-87c1-3e7085bdbd1e"). InnerVolumeSpecName "kube-api-access-gk4hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:52:17.594600 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.594560    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-dshm\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:52:17.594600 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.594597    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-tls-certs\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:52:17.594844 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.594614    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4hr\" (UniqueName: \"kubernetes.io/projected/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e-kube-api-access-gk4hr\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:52:17.795733 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.795695    2563 generic.go:356] "Generic (PLEG): container finished" podID="9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" containerID="4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927" exitCode=0
Jul 24 17:52:17.795733 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.795730    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"
Jul 24 17:52:17.795733 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.795736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv" event={"ID":"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e","Type":"ContainerDied","Data":"4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927"}
Jul 24 17:52:17.796029 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.795776    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv" event={"ID":"9fae2701-ad4f-48f4-87c1-3e7085bdbd1e","Type":"ContainerDied","Data":"f7d8f4e26517e5f7e0090f0d1d4267e650c317114c7336b41f92bc11d936a002"}
Jul 24 17:52:17.796029 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.795796    2563 scope.go:117] "RemoveContainer" containerID="4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927"
Jul 24 17:52:17.806715 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.806690    2563 scope.go:117] "RemoveContainer" containerID="4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927"
Jul 24 17:52:17.807068 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:52:17.807048    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927\": container with ID starting with 4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927 not found: ID does not exist" containerID="4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927"
Jul 24 17:52:17.807149 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.807079    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927"} err="failed to get container status \"4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927\": rpc error: code = NotFound desc = could not find container \"4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927\": container with ID starting with 4ba8a8151f579c159d1eb216e1cba67aabba98ac3bb12b10a632c6200a74c927 not found: ID does not exist"
Jul 24 17:52:17.827294 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.827256    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"]
Jul 24 17:52:17.834481 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:17.834452    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-13c2c7/rate-test-model-13c2c7-kserve-7687fc6c9-jxkhv"]
Jul 24 17:52:18.688516 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:18.688482    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" path="/var/lib/kubelet/pods/9fae2701-ad4f-48f4-87c1-3e7085bdbd1e/volumes"
Jul 24 17:52:23.099207 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:23.099151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:24.240094 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:24.240058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:26.006238 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:26.006180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:26.106857 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:26.106826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:26.410902 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:26.410792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:26.557735 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:26.557701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:27.114542 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.114505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:27.218858 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.218826    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"]
Jul 24 17:52:27.219166 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.219143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:27.219370 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.219354    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" containerName="main"
Jul 24 17:52:27.219421 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.219372    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" containerName="main"
Jul 24 17:52:27.219421 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.219394    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f24ab447-b8fd-4a3e-8542-5154928e0e13" containerName="main"
Jul 24 17:52:27.219421 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.219399    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f24ab447-b8fd-4a3e-8542-5154928e0e13" containerName="main"
Jul 24 17:52:27.219529 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.219472    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fae2701-ad4f-48f4-87c1-3e7085bdbd1e" containerName="main"
Jul 24 17:52:27.219529 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.219483    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f24ab447-b8fd-4a3e-8542-5154928e0e13" containerName="main"
Jul 24 17:52:27.222520 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.222503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:27.224560 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.224544    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-92693f\""
Jul 24 17:52:27.234402 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.234375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"]
Jul 24 17:52:27.245599 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.245553    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"]
Jul 24 17:52:27.253735 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.253705    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:27.257561 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.257531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-92693f-dockercfg-kghd8\""
Jul 24 17:52:27.262976 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.262951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"]
Jul 24 17:52:27.383896 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.383806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzgqg\" (UniqueName: \"kubernetes.io/projected/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-kube-api-access-xzgqg\") pod \"payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf\" (UID: \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:27.383896 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.383854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a762357-028f-4766-a7e9-f96067ed0a1b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr\" (UID: \"5a762357-028f-4766-a7e9-f96067ed0a1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:27.384083 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.383979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z58kh\" (UniqueName: \"kubernetes.io/projected/5a762357-028f-4766-a7e9-f96067ed0a1b-kube-api-access-z58kh\") pod \"payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr\" (UID: \"5a762357-028f-4766-a7e9-f96067ed0a1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:27.384083 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.384005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf\" (UID: \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:27.484742 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.484707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzgqg\" (UniqueName: \"kubernetes.io/projected/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-kube-api-access-xzgqg\") pod \"payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf\" (UID: \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:27.484933 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.484756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a762357-028f-4766-a7e9-f96067ed0a1b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr\" (UID: \"5a762357-028f-4766-a7e9-f96067ed0a1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:27.484933 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.484823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z58kh\" (UniqueName: \"kubernetes.io/projected/5a762357-028f-4766-a7e9-f96067ed0a1b-kube-api-access-z58kh\") pod \"payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr\" (UID: \"5a762357-028f-4766-a7e9-f96067ed0a1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:27.484933 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.484839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf\" (UID: \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:27.485465 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.485448    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf\" (UID: \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:27.485549 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.485531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a762357-028f-4766-a7e9-f96067ed0a1b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr\" (UID: \"5a762357-028f-4766-a7e9-f96067ed0a1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:27.493546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.493507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzgqg\" (UniqueName: \"kubernetes.io/projected/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-kube-api-access-xzgqg\") pod \"payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf\" (UID: \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:27.494290 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.494271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z58kh\" (UniqueName: \"kubernetes.io/projected/5a762357-028f-4766-a7e9-f96067ed0a1b-kube-api-access-z58kh\") pod \"payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr\" (UID: \"5a762357-028f-4766-a7e9-f96067ed0a1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:27.533177 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.533138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:27.566054 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.566018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:27.692465 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.692435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"]
Jul 24 17:52:27.695387 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:52:27.695345    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6179a50_ea54_40ef_83a9_fcbe593b6d5e.slice/crio-55efa1e3819d9c99f5e08e99a1a15b2cf64d05ca96fe5ecdef8afba768b3eccd WatchSource:0}: Error finding container 55efa1e3819d9c99f5e08e99a1a15b2cf64d05ca96fe5ecdef8afba768b3eccd: Status 404 returned error can't find the container with id 55efa1e3819d9c99f5e08e99a1a15b2cf64d05ca96fe5ecdef8afba768b3eccd
Jul 24 17:52:27.724618 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.724595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"]
Jul 24 17:52:27.726974 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:52:27.726948    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a762357_028f_4766_a7e9_f96067ed0a1b.slice/crio-e1280a1db56ccf01492c213506356fdfbfad1e40af75d38eb8ef010bbf5b012f WatchSource:0}: Error finding container e1280a1db56ccf01492c213506356fdfbfad1e40af75d38eb8ef010bbf5b012f: Status 404 returned error can't find the container with id e1280a1db56ccf01492c213506356fdfbfad1e40af75d38eb8ef010bbf5b012f
Jul 24 17:52:27.846054 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.846016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr" event={"ID":"5a762357-028f-4766-a7e9-f96067ed0a1b","Type":"ContainerStarted","Data":"e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b"}
Jul 24 17:52:27.846259 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.846080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr" event={"ID":"5a762357-028f-4766-a7e9-f96067ed0a1b","Type":"ContainerStarted","Data":"e1280a1db56ccf01492c213506356fdfbfad1e40af75d38eb8ef010bbf5b012f"}
Jul 24 17:52:27.846259 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.846213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:27.847662 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.847623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf" event={"ID":"a6179a50-ea54-40ef-83a9-fcbe593b6d5e","Type":"ContainerStarted","Data":"3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6"}
Jul 24 17:52:27.847792 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.847709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:27.847792 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.847728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf" event={"ID":"a6179a50-ea54-40ef-83a9-fcbe593b6d5e","Type":"ContainerStarted","Data":"55efa1e3819d9c99f5e08e99a1a15b2cf64d05ca96fe5ecdef8afba768b3eccd"}
Jul 24 17:52:27.865208 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.865164    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr" podStartSLOduration=0.865148688 podStartE2EDuration="865.148688ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:27.86493826 +0000 UTC m=+2951.696190950" watchObservedRunningTime="2026-07-24 17:52:27.865148688 +0000 UTC m=+2951.696401375"
Jul 24 17:52:27.889132 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:27.889074    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf" podStartSLOduration=0.889055642 podStartE2EDuration="889.055642ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:27.887154967 +0000 UTC m=+2951.718407654" watchObservedRunningTime="2026-07-24 17:52:27.889055642 +0000 UTC m=+2951.720308328"
Jul 24 17:52:30.496530 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:30.496497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:38.854629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:38.854549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:52:38.854629 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:38.854625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:52:39.043595 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:39.043549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:41.445105 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:41.445059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:41.799041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:41.799004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:42.461586 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:42.461534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:44.609071 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:44.609043    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:44.705526 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:44.705493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:44.911997 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:44.911882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:45.191917 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.191834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:45.517124 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.517096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:45.609609 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.609557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:45.652679 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.652614    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"]
Jul 24 17:52:45.656924 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.656898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:52:45.658872 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.658845    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"]
Jul 24 17:52:45.659954 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.659930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ea8b57-dockercfg-pfccq\""
Jul 24 17:52:45.660072 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.659989    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ea8b57\""
Jul 24 17:52:45.662470 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.662452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:45.666760 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.666723    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"]
Jul 24 17:52:45.675884 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.675851    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"]
Jul 24 17:52:45.753599 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.753555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkklf\" (UniqueName: \"kubernetes.io/projected/8db68d3e-5527-4e19-9599-d47b1eaff4a2-kube-api-access-bkklf\") pod \"payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg\" (UID: \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:52:45.753802 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.753618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs\" (UID: \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:45.753802 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.753691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db68d3e-5527-4e19-9599-d47b1eaff4a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg\" (UID: \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:52:45.753802 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.753796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2478\" (UniqueName: \"kubernetes.io/projected/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-kube-api-access-x2478\") pod \"payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs\" (UID: \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:45.854562 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.854465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2478\" (UniqueName: \"kubernetes.io/projected/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-kube-api-access-x2478\") pod \"payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs\" (UID: \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:45.854562 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.854516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkklf\" (UniqueName: \"kubernetes.io/projected/8db68d3e-5527-4e19-9599-d47b1eaff4a2-kube-api-access-bkklf\") pod \"payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg\" (UID: \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:52:45.854562 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.854546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs\" (UID: \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:45.854831 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.854579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db68d3e-5527-4e19-9599-d47b1eaff4a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg\" (UID: \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:52:45.855327 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.855309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db68d3e-5527-4e19-9599-d47b1eaff4a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg\" (UID: \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:52:45.855368 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.855317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs\" (UID: \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:45.864702 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.864640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkklf\" (UniqueName: \"kubernetes.io/projected/8db68d3e-5527-4e19-9599-d47b1eaff4a2-kube-api-access-bkklf\") pod \"payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg\" (UID: \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:52:45.864702 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.864692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2478\" (UniqueName: \"kubernetes.io/projected/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-kube-api-access-x2478\") pod \"payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs\" (UID: \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:45.971338 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.971297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:52:45.985055 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:45.985032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:46.129972 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.129605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"]
Jul 24 17:52:46.133452 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:52:46.133405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db68d3e_5527_4e19_9599_d47b1eaff4a2.slice/crio-95a339a71a0f17f9ebfdf08a53ac161cb51629c09669190dcb5e0bbc0f890b66 WatchSource:0}: Error finding container 95a339a71a0f17f9ebfdf08a53ac161cb51629c09669190dcb5e0bbc0f890b66: Status 404 returned error can't find the container with id 95a339a71a0f17f9ebfdf08a53ac161cb51629c09669190dcb5e0bbc0f890b66
Jul 24 17:52:46.158800 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.158772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"]
Jul 24 17:52:46.173929 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:52:46.173902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1160d2ea_0c10_4d88_b5c9_295a92e6f71f.slice/crio-4e382bcad0ae20bf5e3d49cb0c4fb59116cb8777cb9103879ec52353c91c5637 WatchSource:0}: Error finding container 4e382bcad0ae20bf5e3d49cb0c4fb59116cb8777cb9103879ec52353c91c5637: Status 404 returned error can't find the container with id 4e382bcad0ae20bf5e3d49cb0c4fb59116cb8777cb9103879ec52353c91c5637
Jul 24 17:52:46.948255 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.948218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg" event={"ID":"8db68d3e-5527-4e19-9599-d47b1eaff4a2","Type":"ContainerStarted","Data":"9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c"}
Jul 24 17:52:46.948815 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.948279    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:52:46.948815 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.948305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg" event={"ID":"8db68d3e-5527-4e19-9599-d47b1eaff4a2","Type":"ContainerStarted","Data":"95a339a71a0f17f9ebfdf08a53ac161cb51629c09669190dcb5e0bbc0f890b66"}
Jul 24 17:52:46.949630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.949602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs" event={"ID":"1160d2ea-0c10-4d88-b5c9-295a92e6f71f","Type":"ContainerStarted","Data":"9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4"}
Jul 24 17:52:46.949774 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.949664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs" event={"ID":"1160d2ea-0c10-4d88-b5c9-295a92e6f71f","Type":"ContainerStarted","Data":"4e382bcad0ae20bf5e3d49cb0c4fb59116cb8777cb9103879ec52353c91c5637"}
Jul 24 17:52:46.949774 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.949716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:46.967401 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.967345    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg" podStartSLOduration=1.967329931 podStartE2EDuration="1.967329931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:46.965512207 +0000 UTC m=+2970.796764895" watchObservedRunningTime="2026-07-24 17:52:46.967329931 +0000 UTC m=+2970.798582618"
Jul 24 17:52:46.981761 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:46.981711    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs" podStartSLOduration=1.9816957099999999 podStartE2EDuration="1.98169571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:46.981196551 +0000 UTC m=+2970.812449239" watchObservedRunningTime="2026-07-24 17:52:46.98169571 +0000 UTC m=+2970.812948397"
Jul 24 17:52:49.121188 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:49.121152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:52:57.957221 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:57.957190    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:52:57.957595 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:52:57.957350    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:53:06.487307 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:06.487266    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8n49r_9af7f2dd-29f5-43b5-964f-95142729e455/payload-processing/0.log"
Jul 24 17:53:06.613605 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:06.613561    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg_8db68d3e-5527-4e19-9599-d47b1eaff4a2/payload-processing/0.log"
Jul 24 17:53:16.820630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:16.820604    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:53:16.826923 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:16.826901    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjnf5_62539bf7-8276-46a7-a490-5ede01496cf4/ovn-acl-logging/0.log"
Jul 24 17:53:21.750884 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.750845    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"]
Jul 24 17:53:21.755911 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.755892    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.759105 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.759067    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-92693f\"/\"openshift-service-ca.crt\""
Jul 24 17:53:21.759233 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.759103    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-92693f\"/\"default-dockercfg-6mbtn\""
Jul 24 17:53:21.759233 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.759129    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-92693f\"/\"kube-root-ca.crt\""
Jul 24 17:53:21.759233 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.759140    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-92693f\"/\"ipp-route-92693f-kserve-self-signed-certs\""
Jul 24 17:53:21.764375 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.763951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"]
Jul 24 17:53:21.780479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.780444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:21.791553 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.791517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tls-certs\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.791723 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.791564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s6wd\" (UniqueName: \"kubernetes.io/projected/2be3ba82-3ec8-44df-9f16-1285ee1473c0-kube-api-access-5s6wd\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.791723 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.791631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-home\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.791723 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.791686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-dshm\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.791876 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.791730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-model-cache\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.791876 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.791791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tmp-dir\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.893002 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.892964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tls-certs\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.893196 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.893011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s6wd\" (UniqueName: \"kubernetes.io/projected/2be3ba82-3ec8-44df-9f16-1285ee1473c0-kube-api-access-5s6wd\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.893196 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.893036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-home\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.893196 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.893054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-dshm\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.893196 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.893076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-model-cache\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.893196 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.893113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tmp-dir\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.893511 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.893491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-model-cache\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.893579 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.893540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tmp-dir\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.893622 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.893577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-home\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.895502 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.895483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-dshm\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.895685 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.895668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tls-certs\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:21.901555 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:21.901531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s6wd\" (UniqueName: \"kubernetes.io/projected/2be3ba82-3ec8-44df-9f16-1285ee1473c0-kube-api-access-5s6wd\") pod \"ipp-route-92693f-kserve-59dd767f55-4ttzt\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") " pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:22.066773 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:22.066638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:22.204297 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:22.204248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"]
Jul 24 17:53:22.206543 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:53:22.206514    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2be3ba82_3ec8_44df_9f16_1285ee1473c0.slice/crio-b49151ff316ccb7cde5163df6b54293dc3de45cfd8306595aca3ba780075aedd WatchSource:0}: Error finding container b49151ff316ccb7cde5163df6b54293dc3de45cfd8306595aca3ba780075aedd: Status 404 returned error can't find the container with id b49151ff316ccb7cde5163df6b54293dc3de45cfd8306595aca3ba780075aedd
Jul 24 17:53:23.133989 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:23.133945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt" event={"ID":"2be3ba82-3ec8-44df-9f16-1285ee1473c0","Type":"ContainerStarted","Data":"7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5"}
Jul 24 17:53:23.134382 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:23.133999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt" event={"ID":"2be3ba82-3ec8-44df-9f16-1285ee1473c0","Type":"ContainerStarted","Data":"b49151ff316ccb7cde5163df6b54293dc3de45cfd8306595aca3ba780075aedd"}
Jul 24 17:53:23.151754 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:23.151709    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt" podStartSLOduration=2.15169363 podStartE2EDuration="2.15169363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:53:23.149916634 +0000 UTC m=+3006.981169325" watchObservedRunningTime="2026-07-24 17:53:23.15169363 +0000 UTC m=+3006.982946317"
Jul 24 17:53:32.067632 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:32.067589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:32.068108 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:32.067701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:32.080438 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:32.080405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:32.188932 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:32.188907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:53:53.567334 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:53.567304    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr_5a762357-028f-4766-a7e9-f96067ed0a1b/payload-processing/0.log"
Jul 24 17:53:53.701185 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:53.701145    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8n49r_9af7f2dd-29f5-43b5-964f-95142729e455/payload-processing/0.log"
Jul 24 17:53:53.842182 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:53.842098    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg_8db68d3e-5527-4e19-9599-d47b1eaff4a2/payload-processing/0.log"
Jul 24 17:53:54.189951 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:54.189865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:54.551516 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:54.551479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:54.651224 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:54.651188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:58.057638 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:58.057608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:58.165001 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:58.164957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:58.397370 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:58.397285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:59.266801 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.266768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:59.384289 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.384250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:53:59.489864 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.489827    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"]
Jul 24 17:53:59.493480 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.493464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:53:59.497668 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.497621    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-201d60-dockercfg-z5dd9\""
Jul 24 17:53:59.497857 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.497840    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-201d60\""
Jul 24 17:53:59.514485 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.514459    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"]
Jul 24 17:53:59.537010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.536930    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"]
Jul 24 17:53:59.539474 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.539442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4df\" (UniqueName: \"kubernetes.io/projected/f12e0982-782b-465b-a2d2-4037b1ecfb8c-kube-api-access-tq4df\") pod \"payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr\" (UID: \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:53:59.539556 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.539529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e0982-782b-465b-a2d2-4037b1ecfb8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr\" (UID: \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:53:59.540603 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.540588    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:53:59.554852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.554823    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"]
Jul 24 17:53:59.641010 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.640976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e0982-782b-465b-a2d2-4037b1ecfb8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr\" (UID: \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:53:59.641207 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.641065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxsq\" (UniqueName: \"kubernetes.io/projected/b7c449be-7fd7-44dd-b8fe-992a624f218d-kube-api-access-gwxsq\") pod \"payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r\" (UID: \"b7c449be-7fd7-44dd-b8fe-992a624f218d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:53:59.641207 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.641114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4df\" (UniqueName: \"kubernetes.io/projected/f12e0982-782b-465b-a2d2-4037b1ecfb8c-kube-api-access-tq4df\") pod \"payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr\" (UID: \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:53:59.641207 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.641143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c449be-7fd7-44dd-b8fe-992a624f218d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r\" (UID: \"b7c449be-7fd7-44dd-b8fe-992a624f218d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:53:59.641762 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.641737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e0982-782b-465b-a2d2-4037b1ecfb8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr\" (UID: \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:53:59.652169 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.652143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4df\" (UniqueName: \"kubernetes.io/projected/f12e0982-782b-465b-a2d2-4037b1ecfb8c-kube-api-access-tq4df\") pod \"payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr\" (UID: \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:53:59.741770 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.741726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxsq\" (UniqueName: \"kubernetes.io/projected/b7c449be-7fd7-44dd-b8fe-992a624f218d-kube-api-access-gwxsq\") pod \"payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r\" (UID: \"b7c449be-7fd7-44dd-b8fe-992a624f218d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:53:59.741953 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.741818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c449be-7fd7-44dd-b8fe-992a624f218d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r\" (UID: \"b7c449be-7fd7-44dd-b8fe-992a624f218d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:53:59.742584 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.742561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c449be-7fd7-44dd-b8fe-992a624f218d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r\" (UID: \"b7c449be-7fd7-44dd-b8fe-992a624f218d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:53:59.761858 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.761827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxsq\" (UniqueName: \"kubernetes.io/projected/b7c449be-7fd7-44dd-b8fe-992a624f218d-kube-api-access-gwxsq\") pod \"payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r\" (UID: \"b7c449be-7fd7-44dd-b8fe-992a624f218d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:53:59.804729 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.804614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:53:59.850264 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.850230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:53:59.978294 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:53:59.978264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"]
Jul 24 17:53:59.980980 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:53:59.980948    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12e0982_782b_465b_a2d2_4037b1ecfb8c.slice/crio-c60385a5c430e0e1f68711b8113dee2b4ee0927d84e3ca1a64fef6148b83a7b4 WatchSource:0}: Error finding container c60385a5c430e0e1f68711b8113dee2b4ee0927d84e3ca1a64fef6148b83a7b4: Status 404 returned error can't find the container with id c60385a5c430e0e1f68711b8113dee2b4ee0927d84e3ca1a64fef6148b83a7b4
Jul 24 17:54:00.037003 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:00.036973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"]
Jul 24 17:54:00.043803 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:54:00.043781    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c449be_7fd7_44dd_b8fe_992a624f218d.slice/crio-12357c1a7454ed8aba992802bbb19b55fb505d4525822060d75143e128433265 WatchSource:0}: Error finding container 12357c1a7454ed8aba992802bbb19b55fb505d4525822060d75143e128433265: Status 404 returned error can't find the container with id 12357c1a7454ed8aba992802bbb19b55fb505d4525822060d75143e128433265
Jul 24 17:54:00.322659 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:00.322542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r" event={"ID":"b7c449be-7fd7-44dd-b8fe-992a624f218d","Type":"ContainerStarted","Data":"931ff22bb46fdf1ac949ee5a8316f06d3a9b80d44e7bb96af2093605be86803c"}
Jul 24 17:54:00.322659 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:00.322603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r" event={"ID":"b7c449be-7fd7-44dd-b8fe-992a624f218d","Type":"ContainerStarted","Data":"12357c1a7454ed8aba992802bbb19b55fb505d4525822060d75143e128433265"}
Jul 24 17:54:00.323158 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:00.322702    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:54:00.324271 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:00.324250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr" event={"ID":"f12e0982-782b-465b-a2d2-4037b1ecfb8c","Type":"ContainerStarted","Data":"dc1529def6cf19727b47b3a7cdf59535d25eaece5742270555352289c263e254"}
Jul 24 17:54:00.324401 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:00.324287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr" event={"ID":"f12e0982-782b-465b-a2d2-4037b1ecfb8c","Type":"ContainerStarted","Data":"c60385a5c430e0e1f68711b8113dee2b4ee0927d84e3ca1a64fef6148b83a7b4"}
Jul 24 17:54:00.324401 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:00.324372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:54:00.352686 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:00.352615    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r" podStartSLOduration=1.352599284 podStartE2EDuration="1.352599284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:54:00.350630495 +0000 UTC m=+3044.181883183" watchObservedRunningTime="2026-07-24 17:54:00.352599284 +0000 UTC m=+3044.183851971"
Jul 24 17:54:00.376028 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:00.375977    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr" podStartSLOduration=1.375961171 podStartE2EDuration="1.375961171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:54:00.375875726 +0000 UTC m=+3044.207128413" watchObservedRunningTime="2026-07-24 17:54:00.375961171 +0000 UTC m=+3044.207213857"
Jul 24 17:54:02.575509 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:02.575477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:11.332060 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:11.332022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:54:11.332418 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:11.332209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:54:13.562156 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:13.562122    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"]
Jul 24 17:54:13.566316 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:13.566298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"
Jul 24 17:54:13.569667 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:13.569621    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8llqd\""
Jul 24 17:54:13.581366 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:13.581339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"]
Jul 24 17:54:13.672568 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:13.672533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4cb17-b53b-4a24-88c8-da175d8dcfa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c\" (UID: \"8ef4cb17-b53b-4a24-88c8-da175d8dcfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"
Jul 24 17:54:13.773608 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:13.773570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4cb17-b53b-4a24-88c8-da175d8dcfa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c\" (UID: \"8ef4cb17-b53b-4a24-88c8-da175d8dcfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"
Jul 24 17:54:13.774225 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:13.774207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4cb17-b53b-4a24-88c8-da175d8dcfa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c\" (UID: \"8ef4cb17-b53b-4a24-88c8-da175d8dcfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"
Jul 24 17:54:13.877861 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:13.877776    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"
Jul 24 17:54:14.021229 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:14.021203    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"]
Jul 24 17:54:14.025973 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:54:14.025939    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef4cb17_b53b_4a24_88c8_da175d8dcfa7.slice/crio-beb6d5e6c5b220df131afdde09b75d7f4b2cfa0b36458e1a3de216884156cb76 WatchSource:0}: Error finding container beb6d5e6c5b220df131afdde09b75d7f4b2cfa0b36458e1a3de216884156cb76: Status 404 returned error can't find the container with id beb6d5e6c5b220df131afdde09b75d7f4b2cfa0b36458e1a3de216884156cb76
Jul 24 17:54:14.397359 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:14.397318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c" event={"ID":"8ef4cb17-b53b-4a24-88c8-da175d8dcfa7","Type":"ContainerStarted","Data":"318dd08e042b280d1530b0c6b806e0a73c64971f0ff58a0b0eab39545ddf8d2a"}
Jul 24 17:54:14.397575 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:14.397389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c" event={"ID":"8ef4cb17-b53b-4a24-88c8-da175d8dcfa7","Type":"ContainerStarted","Data":"beb6d5e6c5b220df131afdde09b75d7f4b2cfa0b36458e1a3de216884156cb76"}
Jul 24 17:54:14.415599 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:14.415549    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c" podStartSLOduration=1.415531948 podStartE2EDuration="1.415531948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:54:14.413202353 +0000 UTC m=+3058.244455041" watchObservedRunningTime="2026-07-24 17:54:14.415531948 +0000 UTC m=+3058.246784635"
Jul 24 17:54:15.405940 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:15.405899    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ef4cb17-b53b-4a24-88c8-da175d8dcfa7" containerID="318dd08e042b280d1530b0c6b806e0a73c64971f0ff58a0b0eab39545ddf8d2a" exitCode=0
Jul 24 17:54:15.406362 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:15.405978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c" event={"ID":"8ef4cb17-b53b-4a24-88c8-da175d8dcfa7","Type":"ContainerDied","Data":"318dd08e042b280d1530b0c6b806e0a73c64971f0ff58a0b0eab39545ddf8d2a"}
Jul 24 17:54:16.561243 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:16.561218    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"
Jul 24 17:54:16.702245 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:16.702167    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4cb17-b53b-4a24-88c8-da175d8dcfa7-maas-api-service-ca\") pod \"8ef4cb17-b53b-4a24-88c8-da175d8dcfa7\" (UID: \"8ef4cb17-b53b-4a24-88c8-da175d8dcfa7\") "
Jul 24 17:54:16.702503 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:16.702480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef4cb17-b53b-4a24-88c8-da175d8dcfa7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ef4cb17-b53b-4a24-88c8-da175d8dcfa7" (UID: "8ef4cb17-b53b-4a24-88c8-da175d8dcfa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:54:16.803286 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:16.803255    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef4cb17-b53b-4a24-88c8-da175d8dcfa7-maas-api-service-ca\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:54:17.418276 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:17.418246    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"
Jul 24 17:54:17.418276 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:17.418255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c" event={"ID":"8ef4cb17-b53b-4a24-88c8-da175d8dcfa7","Type":"ContainerDied","Data":"beb6d5e6c5b220df131afdde09b75d7f4b2cfa0b36458e1a3de216884156cb76"}
Jul 24 17:54:17.418491 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:17.418299    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beb6d5e6c5b220df131afdde09b75d7f4b2cfa0b36458e1a3de216884156cb76"
Jul 24 17:54:22.844984 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:22.844950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:24.211933 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.211900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:24.256758 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.256716    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"]
Jul 24 17:54:24.257042 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.256990    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r" podUID="b7c449be-7fd7-44dd-b8fe-992a624f218d" containerName="payload-processing" containerID="cri-o://931ff22bb46fdf1ac949ee5a8316f06d3a9b80d44e7bb96af2093605be86803c" gracePeriod=30
Jul 24 17:54:24.265009 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.264891    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"]
Jul 24 17:54:24.265188 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.265122    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr" podUID="f12e0982-782b-465b-a2d2-4037b1ecfb8c" containerName="payload-pre-processing" containerID="cri-o://dc1529def6cf19727b47b3a7cdf59535d25eaece5742270555352289c263e254" gracePeriod=30
Jul 24 17:54:24.309678 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.307388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:24.378993 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.378952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:24.456808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.456774    2563 generic.go:356] "Generic (PLEG): container finished" podID="b7c449be-7fd7-44dd-b8fe-992a624f218d" containerID="931ff22bb46fdf1ac949ee5a8316f06d3a9b80d44e7bb96af2093605be86803c" exitCode=0
Jul 24 17:54:24.457052 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.456849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r" event={"ID":"b7c449be-7fd7-44dd-b8fe-992a624f218d","Type":"ContainerDied","Data":"931ff22bb46fdf1ac949ee5a8316f06d3a9b80d44e7bb96af2093605be86803c"}
Jul 24 17:54:24.458821 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.458788    2563 generic.go:356] "Generic (PLEG): container finished" podID="f12e0982-782b-465b-a2d2-4037b1ecfb8c" containerID="dc1529def6cf19727b47b3a7cdf59535d25eaece5742270555352289c263e254" exitCode=0
Jul 24 17:54:24.458941 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.458828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr" event={"ID":"f12e0982-782b-465b-a2d2-4037b1ecfb8c","Type":"ContainerDied","Data":"dc1529def6cf19727b47b3a7cdf59535d25eaece5742270555352289c263e254"}
Jul 24 17:54:24.524529 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.524501    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:54:24.559935 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.559903    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:54:24.574344 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.574311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwxsq\" (UniqueName: \"kubernetes.io/projected/b7c449be-7fd7-44dd-b8fe-992a624f218d-kube-api-access-gwxsq\") pod \"b7c449be-7fd7-44dd-b8fe-992a624f218d\" (UID: \"b7c449be-7fd7-44dd-b8fe-992a624f218d\") "
Jul 24 17:54:24.574495 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.574371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c449be-7fd7-44dd-b8fe-992a624f218d-plugins-config-volume\") pod \"b7c449be-7fd7-44dd-b8fe-992a624f218d\" (UID: \"b7c449be-7fd7-44dd-b8fe-992a624f218d\") "
Jul 24 17:54:24.574786 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.574759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c449be-7fd7-44dd-b8fe-992a624f218d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7c449be-7fd7-44dd-b8fe-992a624f218d" (UID: "b7c449be-7fd7-44dd-b8fe-992a624f218d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:54:24.576541 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.576517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c449be-7fd7-44dd-b8fe-992a624f218d-kube-api-access-gwxsq" (OuterVolumeSpecName: "kube-api-access-gwxsq") pod "b7c449be-7fd7-44dd-b8fe-992a624f218d" (UID: "b7c449be-7fd7-44dd-b8fe-992a624f218d"). InnerVolumeSpecName "kube-api-access-gwxsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:54:24.675318 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.675282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq4df\" (UniqueName: \"kubernetes.io/projected/f12e0982-782b-465b-a2d2-4037b1ecfb8c-kube-api-access-tq4df\") pod \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\" (UID: \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\") "
Jul 24 17:54:24.675490 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.675368    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e0982-782b-465b-a2d2-4037b1ecfb8c-plugins-config-volume\") pod \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\" (UID: \"f12e0982-782b-465b-a2d2-4037b1ecfb8c\") "
Jul 24 17:54:24.675706 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.675689    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c449be-7fd7-44dd-b8fe-992a624f218d-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:54:24.675782 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.675711    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxsq\" (UniqueName: \"kubernetes.io/projected/b7c449be-7fd7-44dd-b8fe-992a624f218d-kube-api-access-gwxsq\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:54:24.675782 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.675729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12e0982-782b-465b-a2d2-4037b1ecfb8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f12e0982-782b-465b-a2d2-4037b1ecfb8c" (UID: "f12e0982-782b-465b-a2d2-4037b1ecfb8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:54:24.677469 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.677447    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12e0982-782b-465b-a2d2-4037b1ecfb8c-kube-api-access-tq4df" (OuterVolumeSpecName: "kube-api-access-tq4df") pod "f12e0982-782b-465b-a2d2-4037b1ecfb8c" (UID: "f12e0982-782b-465b-a2d2-4037b1ecfb8c"). InnerVolumeSpecName "kube-api-access-tq4df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:54:24.776597 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.776503    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4df\" (UniqueName: \"kubernetes.io/projected/f12e0982-782b-465b-a2d2-4037b1ecfb8c-kube-api-access-tq4df\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:54:24.776597 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:24.776536    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e0982-782b-465b-a2d2-4037b1ecfb8c-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:54:25.465367 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.465329    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"
Jul 24 17:54:25.465367 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.465348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r" event={"ID":"b7c449be-7fd7-44dd-b8fe-992a624f218d","Type":"ContainerDied","Data":"12357c1a7454ed8aba992802bbb19b55fb505d4525822060d75143e128433265"}
Jul 24 17:54:25.465925 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.465426    2563 scope.go:117] "RemoveContainer" containerID="931ff22bb46fdf1ac949ee5a8316f06d3a9b80d44e7bb96af2093605be86803c"
Jul 24 17:54:25.466852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.466828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr" event={"ID":"f12e0982-782b-465b-a2d2-4037b1ecfb8c","Type":"ContainerDied","Data":"c60385a5c430e0e1f68711b8113dee2b4ee0927d84e3ca1a64fef6148b83a7b4"}
Jul 24 17:54:25.466977 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.466833    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"
Jul 24 17:54:25.474881 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.474817    2563 scope.go:117] "RemoveContainer" containerID="dc1529def6cf19727b47b3a7cdf59535d25eaece5742270555352289c263e254"
Jul 24 17:54:25.485933 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.485909    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"]
Jul 24 17:54:25.492388 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.492367    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-201d60-57f647fc58-g549r"]
Jul 24 17:54:25.529382 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.529282    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"]
Jul 24 17:54:25.537594 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:25.537563    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-201d60-6454dff6d8-kbcfr"]
Jul 24 17:54:26.688092 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:26.688058    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c449be-7fd7-44dd-b8fe-992a624f218d" path="/var/lib/kubelet/pods/b7c449be-7fd7-44dd-b8fe-992a624f218d/volumes"
Jul 24 17:54:26.688464 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:26.688390    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12e0982-782b-465b-a2d2-4037b1ecfb8c" path="/var/lib/kubelet/pods/f12e0982-782b-465b-a2d2-4037b1ecfb8c/volumes"
Jul 24 17:54:28.645184 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:28.645141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:28.645579 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:28.645544    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"]
Jul 24 17:54:28.649059 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:28.649038    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-201d60-t669c"]
Jul 24 17:54:28.688472 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:28.688443    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ef4cb17-b53b-4a24-88c8-da175d8dcfa7" path="/var/lib/kubelet/pods/8ef4cb17-b53b-4a24-88c8-da175d8dcfa7/volumes"
Jul 24 17:54:29.061462 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:29.061431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:35.341254 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:35.341221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:35.439911 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:35.439875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:35.659127 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:35.659024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:35.890724 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:35.890677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:41.649659 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:41.649622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:41.752889 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:41.752854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:55.288442 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.288395    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"]
Jul 24 17:54:55.288993 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.288716    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr" podUID="5a762357-028f-4766-a7e9-f96067ed0a1b" containerName="payload-processing" containerID="cri-o://e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b" gracePeriod=30
Jul 24 17:54:55.313729 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.313699    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"]
Jul 24 17:54:55.313956 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.313933    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf" podUID="a6179a50-ea54-40ef-83a9-fcbe593b6d5e" containerName="payload-pre-processing" containerID="cri-o://3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6" gracePeriod=30
Jul 24 17:54:55.405535 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.405501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:55.520479 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.520446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:54:55.564810 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.564784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:54:55.595251 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.595224    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:54:55.617546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.617512    2563 generic.go:356] "Generic (PLEG): container finished" podID="a6179a50-ea54-40ef-83a9-fcbe593b6d5e" containerID="3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6" exitCode=0
Jul 24 17:54:55.617546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.617556    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"
Jul 24 17:54:55.617852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.617594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf" event={"ID":"a6179a50-ea54-40ef-83a9-fcbe593b6d5e","Type":"ContainerDied","Data":"3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6"}
Jul 24 17:54:55.617852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.617705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf" event={"ID":"a6179a50-ea54-40ef-83a9-fcbe593b6d5e","Type":"ContainerDied","Data":"55efa1e3819d9c99f5e08e99a1a15b2cf64d05ca96fe5ecdef8afba768b3eccd"}
Jul 24 17:54:55.617852 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.617731    2563 scope.go:117] "RemoveContainer" containerID="3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6"
Jul 24 17:54:55.619136 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.619109    2563 generic.go:356] "Generic (PLEG): container finished" podID="5a762357-028f-4766-a7e9-f96067ed0a1b" containerID="e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b" exitCode=0
Jul 24 17:54:55.619136 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.619139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr" event={"ID":"5a762357-028f-4766-a7e9-f96067ed0a1b","Type":"ContainerDied","Data":"e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b"}
Jul 24 17:54:55.619320 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.619191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr" event={"ID":"5a762357-028f-4766-a7e9-f96067ed0a1b","Type":"ContainerDied","Data":"e1280a1db56ccf01492c213506356fdfbfad1e40af75d38eb8ef010bbf5b012f"}
Jul 24 17:54:55.619320 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.619147    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"
Jul 24 17:54:55.628477 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.628447    2563 scope.go:117] "RemoveContainer" containerID="3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6"
Jul 24 17:54:55.628798 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:54:55.628775    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6\": container with ID starting with 3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6 not found: ID does not exist" containerID="3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6"
Jul 24 17:54:55.628886 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.628813    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6"} err="failed to get container status \"3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6\": rpc error: code = NotFound desc = could not find container \"3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6\": container with ID starting with 3ff75c236e4c08eaa79d66d536bdc641aa4876d1c4a1378212be4a28343310c6 not found: ID does not exist"
Jul 24 17:54:55.628886 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.628839    2563 scope.go:117] "RemoveContainer" containerID="e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b"
Jul 24 17:54:55.644468 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.644448    2563 scope.go:117] "RemoveContainer" containerID="e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b"
Jul 24 17:54:55.644810 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:54:55.644792    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b\": container with ID starting with e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b not found: ID does not exist" containerID="e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b"
Jul 24 17:54:55.644873 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.644819    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b"} err="failed to get container status \"e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b\": rpc error: code = NotFound desc = could not find container \"e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b\": container with ID starting with e6cf715ed118927579d18b0b456367414df8b6f5f4df3ffc73da3c0d04e3837b not found: ID does not exist"
Jul 24 17:54:55.660324 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.660289    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-plugins-config-volume\") pod \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\" (UID: \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\") "
Jul 24 17:54:55.660456 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.660354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a762357-028f-4766-a7e9-f96067ed0a1b-plugins-config-volume\") pod \"5a762357-028f-4766-a7e9-f96067ed0a1b\" (UID: \"5a762357-028f-4766-a7e9-f96067ed0a1b\") "
Jul 24 17:54:55.660456 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.660393    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzgqg\" (UniqueName: \"kubernetes.io/projected/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-kube-api-access-xzgqg\") pod \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\" (UID: \"a6179a50-ea54-40ef-83a9-fcbe593b6d5e\") "
Jul 24 17:54:55.660549 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.660493    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z58kh\" (UniqueName: \"kubernetes.io/projected/5a762357-028f-4766-a7e9-f96067ed0a1b-kube-api-access-z58kh\") pod \"5a762357-028f-4766-a7e9-f96067ed0a1b\" (UID: \"5a762357-028f-4766-a7e9-f96067ed0a1b\") "
Jul 24 17:54:55.660802 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.660778    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6179a50-ea54-40ef-83a9-fcbe593b6d5e" (UID: "a6179a50-ea54-40ef-83a9-fcbe593b6d5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:54:55.660858 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.660803    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a762357-028f-4766-a7e9-f96067ed0a1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a762357-028f-4766-a7e9-f96067ed0a1b" (UID: "5a762357-028f-4766-a7e9-f96067ed0a1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:54:55.660932 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.660912    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:54:55.660999 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.660940    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a762357-028f-4766-a7e9-f96067ed0a1b-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:54:55.662819 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.662793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-kube-api-access-xzgqg" (OuterVolumeSpecName: "kube-api-access-xzgqg") pod "a6179a50-ea54-40ef-83a9-fcbe593b6d5e" (UID: "a6179a50-ea54-40ef-83a9-fcbe593b6d5e"). InnerVolumeSpecName "kube-api-access-xzgqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:54:55.662819 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.662805    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a762357-028f-4766-a7e9-f96067ed0a1b-kube-api-access-z58kh" (OuterVolumeSpecName: "kube-api-access-z58kh") pod "5a762357-028f-4766-a7e9-f96067ed0a1b" (UID: "5a762357-028f-4766-a7e9-f96067ed0a1b"). InnerVolumeSpecName "kube-api-access-z58kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:54:55.761949 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.761910    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z58kh\" (UniqueName: \"kubernetes.io/projected/5a762357-028f-4766-a7e9-f96067ed0a1b-kube-api-access-z58kh\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:54:55.761949 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.761942    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzgqg\" (UniqueName: \"kubernetes.io/projected/a6179a50-ea54-40ef-83a9-fcbe593b6d5e-kube-api-access-xzgqg\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:54:55.943566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.943441    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"]
Jul 24 17:54:55.945292 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.945270    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92693f-7458cc6548-kvvrf"]
Jul 24 17:54:55.957736 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.957711    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"]
Jul 24 17:54:55.962902 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:55.962879    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92693f-5cfcdf9596-7gjlr"]
Jul 24 17:54:56.688805 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:56.688725    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a762357-028f-4766-a7e9-f96067ed0a1b" path="/var/lib/kubelet/pods/5a762357-028f-4766-a7e9-f96067ed0a1b/volumes"
Jul 24 17:54:56.689160 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:56.689076    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6179a50-ea54-40ef-83a9-fcbe593b6d5e" path="/var/lib/kubelet/pods/a6179a50-ea54-40ef-83a9-fcbe593b6d5e/volumes"
Jul 24 17:54:59.710610 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:54:59.710576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:00.135367 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:00.135299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:05.107006 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:05.106961    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"]
Jul 24 17:55:05.203735 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:05.203701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:11.916331 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:11.916296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:12.053332 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:12.053296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:20.126844 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.126802    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt" podUID="2be3ba82-3ec8-44df-9f16-1285ee1473c0" containerName="main" containerID="cri-o://7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5" gracePeriod=45
Jul 24 17:55:20.389045 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.388985    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:55:20.490210 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-home\") pod \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") "
Jul 24 17:55:20.490210 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s6wd\" (UniqueName: \"kubernetes.io/projected/2be3ba82-3ec8-44df-9f16-1285ee1473c0-kube-api-access-5s6wd\") pod \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") "
Jul 24 17:55:20.490463 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-dshm\") pod \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") "
Jul 24 17:55:20.490463 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tls-certs\") pod \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") "
Jul 24 17:55:20.490463 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tmp-dir\") pod \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") "
Jul 24 17:55:20.490624 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490463    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-model-cache\") pod \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\" (UID: \"2be3ba82-3ec8-44df-9f16-1285ee1473c0\") "
Jul 24 17:55:20.490624 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490491    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-home" (OuterVolumeSpecName: "home") pod "2be3ba82-3ec8-44df-9f16-1285ee1473c0" (UID: "2be3ba82-3ec8-44df-9f16-1285ee1473c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:55:20.490624 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490585    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2be3ba82-3ec8-44df-9f16-1285ee1473c0" (UID: "2be3ba82-3ec8-44df-9f16-1285ee1473c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:55:20.490840 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "2be3ba82-3ec8-44df-9f16-1285ee1473c0" (UID: "2be3ba82-3ec8-44df-9f16-1285ee1473c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:55:20.490945 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490928    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tmp-dir\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:55:20.490998 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490950    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-model-cache\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:55:20.490998 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.490965    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-home\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:55:20.492815 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.492790    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2be3ba82-3ec8-44df-9f16-1285ee1473c0" (UID: "2be3ba82-3ec8-44df-9f16-1285ee1473c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:55:20.493060 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.493033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-dshm" (OuterVolumeSpecName: "dshm") pod "2be3ba82-3ec8-44df-9f16-1285ee1473c0" (UID: "2be3ba82-3ec8-44df-9f16-1285ee1473c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:55:20.493060 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.493053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be3ba82-3ec8-44df-9f16-1285ee1473c0-kube-api-access-5s6wd" (OuterVolumeSpecName: "kube-api-access-5s6wd") pod "2be3ba82-3ec8-44df-9f16-1285ee1473c0" (UID: "2be3ba82-3ec8-44df-9f16-1285ee1473c0"). InnerVolumeSpecName "kube-api-access-5s6wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:55:20.592448 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.592413    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be3ba82-3ec8-44df-9f16-1285ee1473c0-dshm\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:55:20.592448 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.592445    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2be3ba82-3ec8-44df-9f16-1285ee1473c0-tls-certs\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:55:20.592448 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.592455    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s6wd\" (UniqueName: \"kubernetes.io/projected/2be3ba82-3ec8-44df-9f16-1285ee1473c0-kube-api-access-5s6wd\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:55:20.746993 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.746963    2563 generic.go:356] "Generic (PLEG): container finished" podID="2be3ba82-3ec8-44df-9f16-1285ee1473c0" containerID="7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5" exitCode=0
Jul 24 17:55:20.747184 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.747005    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"
Jul 24 17:55:20.747184 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.747042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt" event={"ID":"2be3ba82-3ec8-44df-9f16-1285ee1473c0","Type":"ContainerDied","Data":"7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5"}
Jul 24 17:55:20.747184 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.747087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt" event={"ID":"2be3ba82-3ec8-44df-9f16-1285ee1473c0","Type":"ContainerDied","Data":"b49151ff316ccb7cde5163df6b54293dc3de45cfd8306595aca3ba780075aedd"}
Jul 24 17:55:20.747184 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.747109    2563 scope.go:117] "RemoveContainer" containerID="7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5"
Jul 24 17:55:20.756224 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.756207    2563 scope.go:117] "RemoveContainer" containerID="7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5"
Jul 24 17:55:20.756518 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:55:20.756476    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5\": container with ID starting with 7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5 not found: ID does not exist" containerID="7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5"
Jul 24 17:55:20.756626 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.756530    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5"} err="failed to get container status \"7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5\": rpc error: code = NotFound desc = could not find container \"7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5\": container with ID starting with 7fd337cc6e67d0454a78f34e5ad727dc4b3ddd49478c907c7fa8a15f6490b6b5 not found: ID does not exist"
Jul 24 17:55:20.765974 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.765948    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"]
Jul 24 17:55:20.773303 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:20.773279    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-92693f/ipp-route-92693f-kserve-59dd767f55-4ttzt"]
Jul 24 17:55:22.688619 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:22.688589    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2be3ba82-3ec8-44df-9f16-1285ee1473c0" path="/var/lib/kubelet/pods/2be3ba82-3ec8-44df-9f16-1285ee1473c0/volumes"
Jul 24 17:55:42.172623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:42.172538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:43.424450 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:43.424409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:49.330486 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:49.330423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:49.435485 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:49.435449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:49.659216 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:49.659139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:49.924147 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:49.924060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:55:50.063770 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.063729    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"]
Jul 24 17:55:50.064217 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064200    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ef4cb17-b53b-4a24-88c8-da175d8dcfa7" containerName="revoke-keys"
Jul 24 17:55:50.064305 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064221    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ef4cb17-b53b-4a24-88c8-da175d8dcfa7" containerName="revoke-keys"
Jul 24 17:55:50.064305 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064238    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6179a50-ea54-40ef-83a9-fcbe593b6d5e" containerName="payload-pre-processing"
Jul 24 17:55:50.064305 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064247    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6179a50-ea54-40ef-83a9-fcbe593b6d5e" containerName="payload-pre-processing"
Jul 24 17:55:50.064305 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064268    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2be3ba82-3ec8-44df-9f16-1285ee1473c0" containerName="main"
Jul 24 17:55:50.064305 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064276    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2be3ba82-3ec8-44df-9f16-1285ee1473c0" containerName="main"
Jul 24 17:55:50.064305 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064297    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c449be-7fd7-44dd-b8fe-992a624f218d" containerName="payload-processing"
Jul 24 17:55:50.064305 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064306    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c449be-7fd7-44dd-b8fe-992a624f218d" containerName="payload-processing"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064319    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a762357-028f-4766-a7e9-f96067ed0a1b" containerName="payload-processing"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064328    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a762357-028f-4766-a7e9-f96067ed0a1b" containerName="payload-processing"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064336    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12e0982-782b-465b-a2d2-4037b1ecfb8c" containerName="payload-pre-processing"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064344    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12e0982-782b-465b-a2d2-4037b1ecfb8c" containerName="payload-pre-processing"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064452    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6179a50-ea54-40ef-83a9-fcbe593b6d5e" containerName="payload-pre-processing"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064464    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2be3ba82-3ec8-44df-9f16-1285ee1473c0" containerName="main"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064472    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ef4cb17-b53b-4a24-88c8-da175d8dcfa7" containerName="revoke-keys"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064482    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a762357-028f-4766-a7e9-f96067ed0a1b" containerName="payload-processing"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064491    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c449be-7fd7-44dd-b8fe-992a624f218d" containerName="payload-processing"
Jul 24 17:55:50.064665 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.064502    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12e0982-782b-465b-a2d2-4037b1ecfb8c" containerName="payload-pre-processing"
Jul 24 17:55:50.067847 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.067827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"
Jul 24 17:55:50.070888 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.070867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8llqd\""
Jul 24 17:55:50.078345 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.078321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"]
Jul 24 17:55:50.153962 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.153929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83206685-93b9-446a-ad4c-cf1c4567672a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq\" (UID: \"83206685-93b9-446a-ad4c-cf1c4567672a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"
Jul 24 17:55:50.255311 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.255260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83206685-93b9-446a-ad4c-cf1c4567672a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq\" (UID: \"83206685-93b9-446a-ad4c-cf1c4567672a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"
Jul 24 17:55:50.255947 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.255928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83206685-93b9-446a-ad4c-cf1c4567672a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq\" (UID: \"83206685-93b9-446a-ad4c-cf1c4567672a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"
Jul 24 17:55:50.378623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.378585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"
Jul 24 17:55:50.520608 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.520584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"]
Jul 24 17:55:50.522885 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:55:50.522821    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83206685_93b9_446a_ad4c_cf1c4567672a.slice/crio-14df9642737ee21d74e8628c040c8d48ca54165e32a89ad2a18031191c26ccbb WatchSource:0}: Error finding container 14df9642737ee21d74e8628c040c8d48ca54165e32a89ad2a18031191c26ccbb: Status 404 returned error can't find the container with id 14df9642737ee21d74e8628c040c8d48ca54165e32a89ad2a18031191c26ccbb
Jul 24 17:55:50.891549 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.891512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq" event={"ID":"83206685-93b9-446a-ad4c-cf1c4567672a","Type":"ContainerStarted","Data":"00942a356e7b464872eaa282127676218c6ed6ad155a441450f534feae096f8e"}
Jul 24 17:55:50.891751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.891569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq" event={"ID":"83206685-93b9-446a-ad4c-cf1c4567672a","Type":"ContainerStarted","Data":"14df9642737ee21d74e8628c040c8d48ca54165e32a89ad2a18031191c26ccbb"}
Jul 24 17:55:50.913608 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:50.913555    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq" podStartSLOduration=0.913540036 podStartE2EDuration="913.540036ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:55:50.911523767 +0000 UTC m=+3154.742776455" watchObservedRunningTime="2026-07-24 17:55:50.913540036 +0000 UTC m=+3154.744792722"
Jul 24 17:55:51.897631 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:51.897591    2563 generic.go:356] "Generic (PLEG): container finished" podID="83206685-93b9-446a-ad4c-cf1c4567672a" containerID="00942a356e7b464872eaa282127676218c6ed6ad155a441450f534feae096f8e" exitCode=0
Jul 24 17:55:51.898104 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:51.897686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq" event={"ID":"83206685-93b9-446a-ad4c-cf1c4567672a","Type":"ContainerDied","Data":"00942a356e7b464872eaa282127676218c6ed6ad155a441450f534feae096f8e"}
Jul 24 17:55:53.041663 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:53.041618    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"
Jul 24 17:55:53.082817 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:53.082784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83206685-93b9-446a-ad4c-cf1c4567672a-maas-api-service-ca\") pod \"83206685-93b9-446a-ad4c-cf1c4567672a\" (UID: \"83206685-93b9-446a-ad4c-cf1c4567672a\") "
Jul 24 17:55:53.083113 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:53.083092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83206685-93b9-446a-ad4c-cf1c4567672a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83206685-93b9-446a-ad4c-cf1c4567672a" (UID: "83206685-93b9-446a-ad4c-cf1c4567672a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:55:53.184064 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:53.183961    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83206685-93b9-446a-ad4c-cf1c4567672a-maas-api-service-ca\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:55:53.911715 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:53.911679    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"
Jul 24 17:55:53.911715 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:53.911697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq" event={"ID":"83206685-93b9-446a-ad4c-cf1c4567672a","Type":"ContainerDied","Data":"14df9642737ee21d74e8628c040c8d48ca54165e32a89ad2a18031191c26ccbb"}
Jul 24 17:55:53.911914 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:55:53.911752    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14df9642737ee21d74e8628c040c8d48ca54165e32a89ad2a18031191c26ccbb"
Jul 24 17:56:00.976000 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:00.975954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:01.004175 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.004140    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"]
Jul 24 17:56:01.004419 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.004392    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg" podUID="8db68d3e-5527-4e19-9599-d47b1eaff4a2" containerName="payload-processing" containerID="cri-o://9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c" gracePeriod=30
Jul 24 17:56:01.024935 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.024905    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"]
Jul 24 17:56:01.025216 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.025168    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs" podUID="1160d2ea-0c10-4d88-b5c9-295a92e6f71f" containerName="payload-pre-processing" containerID="cri-o://9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4" gracePeriod=30
Jul 24 17:56:01.079423 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.079377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:01.177611 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.177579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:01.290602 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.290576    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:56:01.323990 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.323963    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:56:01.359105 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.359076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db68d3e-5527-4e19-9599-d47b1eaff4a2-plugins-config-volume\") pod \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\" (UID: \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\") "
Jul 24 17:56:01.359299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.359130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkklf\" (UniqueName: \"kubernetes.io/projected/8db68d3e-5527-4e19-9599-d47b1eaff4a2-kube-api-access-bkklf\") pod \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\" (UID: \"8db68d3e-5527-4e19-9599-d47b1eaff4a2\") "
Jul 24 17:56:01.359515 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.359491    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db68d3e-5527-4e19-9599-d47b1eaff4a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8db68d3e-5527-4e19-9599-d47b1eaff4a2" (UID: "8db68d3e-5527-4e19-9599-d47b1eaff4a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:56:01.361455 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.361427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db68d3e-5527-4e19-9599-d47b1eaff4a2-kube-api-access-bkklf" (OuterVolumeSpecName: "kube-api-access-bkklf") pod "8db68d3e-5527-4e19-9599-d47b1eaff4a2" (UID: "8db68d3e-5527-4e19-9599-d47b1eaff4a2"). InnerVolumeSpecName "kube-api-access-bkklf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:56:01.460360 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.460321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2478\" (UniqueName: \"kubernetes.io/projected/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-kube-api-access-x2478\") pod \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\" (UID: \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\") "
Jul 24 17:56:01.460543 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.460411    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-plugins-config-volume\") pod \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\" (UID: \"1160d2ea-0c10-4d88-b5c9-295a92e6f71f\") "
Jul 24 17:56:01.460585 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.460555    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkklf\" (UniqueName: \"kubernetes.io/projected/8db68d3e-5527-4e19-9599-d47b1eaff4a2-kube-api-access-bkklf\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:01.460585 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.460566    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db68d3e-5527-4e19-9599-d47b1eaff4a2-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:01.460861 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.460827    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1160d2ea-0c10-4d88-b5c9-295a92e6f71f" (UID: "1160d2ea-0c10-4d88-b5c9-295a92e6f71f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:56:01.462713 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.462688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-kube-api-access-x2478" (OuterVolumeSpecName: "kube-api-access-x2478") pod "1160d2ea-0c10-4d88-b5c9-295a92e6f71f" (UID: "1160d2ea-0c10-4d88-b5c9-295a92e6f71f"). InnerVolumeSpecName "kube-api-access-x2478". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:56:01.561724 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.561613    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2478\" (UniqueName: \"kubernetes.io/projected/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-kube-api-access-x2478\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:01.561724 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.561642    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1160d2ea-0c10-4d88-b5c9-295a92e6f71f-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:01.953345 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.953261    2563 generic.go:356] "Generic (PLEG): container finished" podID="8db68d3e-5527-4e19-9599-d47b1eaff4a2" containerID="9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c" exitCode=0
Jul 24 17:56:01.953345 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.953308    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"
Jul 24 17:56:01.953559 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.953352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg" event={"ID":"8db68d3e-5527-4e19-9599-d47b1eaff4a2","Type":"ContainerDied","Data":"9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c"}
Jul 24 17:56:01.953559 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.953406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg" event={"ID":"8db68d3e-5527-4e19-9599-d47b1eaff4a2","Type":"ContainerDied","Data":"95a339a71a0f17f9ebfdf08a53ac161cb51629c09669190dcb5e0bbc0f890b66"}
Jul 24 17:56:01.953559 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.953440    2563 scope.go:117] "RemoveContainer" containerID="9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c"
Jul 24 17:56:01.954728 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.954704    2563 generic.go:356] "Generic (PLEG): container finished" podID="1160d2ea-0c10-4d88-b5c9-295a92e6f71f" containerID="9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4" exitCode=0
Jul 24 17:56:01.954845 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.954756    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"
Jul 24 17:56:01.954845 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.954774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs" event={"ID":"1160d2ea-0c10-4d88-b5c9-295a92e6f71f","Type":"ContainerDied","Data":"9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4"}
Jul 24 17:56:01.954845 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.954812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs" event={"ID":"1160d2ea-0c10-4d88-b5c9-295a92e6f71f","Type":"ContainerDied","Data":"4e382bcad0ae20bf5e3d49cb0c4fb59116cb8777cb9103879ec52353c91c5637"}
Jul 24 17:56:01.963551 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.963529    2563 scope.go:117] "RemoveContainer" containerID="9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c"
Jul 24 17:56:01.963932 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:56:01.963907    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c\": container with ID starting with 9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c not found: ID does not exist" containerID="9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c"
Jul 24 17:56:01.964018 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.963936    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c"} err="failed to get container status \"9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c\": rpc error: code = NotFound desc = could not find container \"9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c\": container with ID starting with 9b026415c2c6d0d6a0e12b85a3d79533d9412074f5760012f01205d1c735347c not found: ID does not exist"
Jul 24 17:56:01.964018 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.963951    2563 scope.go:117] "RemoveContainer" containerID="9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4"
Jul 24 17:56:01.971898 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.971880    2563 scope.go:117] "RemoveContainer" containerID="9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4"
Jul 24 17:56:01.972205 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:56:01.972186    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4\": container with ID starting with 9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4 not found: ID does not exist" containerID="9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4"
Jul 24 17:56:01.972253 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.972214    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4"} err="failed to get container status \"9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4\": rpc error: code = NotFound desc = could not find container \"9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4\": container with ID starting with 9a8741b9d6290dfbe4e10391f22cefef6e36182164d7acc624d1ab8a14ea9df4 not found: ID does not exist"
Jul 24 17:56:01.986316 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:01.986292    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"]
Jul 24 17:56:02.000362 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:02.000336    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea8b57-5cfd55759b-qwfwg"]
Jul 24 17:56:02.020830 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:02.020807    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"]
Jul 24 17:56:02.029380 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:02.029359    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea8b57-6d64795844-d79bs"]
Jul 24 17:56:02.688829 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:02.688800    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1160d2ea-0c10-4d88-b5c9-295a92e6f71f" path="/var/lib/kubelet/pods/1160d2ea-0c10-4d88-b5c9-295a92e6f71f/volumes"
Jul 24 17:56:02.689159 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:02.689147    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db68d3e-5527-4e19-9599-d47b1eaff4a2" path="/var/lib/kubelet/pods/8db68d3e-5527-4e19-9599-d47b1eaff4a2/volumes"
Jul 24 17:56:05.140666 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:05.140608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:05.146068 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:05.146043    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"]
Jul 24 17:56:05.148223 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:05.148201    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea8b57-wmtcq"]
Jul 24 17:56:05.580153 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:05.580120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:06.688559 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:06.688525    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83206685-93b9-446a-ad4c-cf1c4567672a" path="/var/lib/kubelet/pods/83206685-93b9-446a-ad4c-cf1c4567672a/volumes"
Jul 24 17:56:09.764048 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:09.764018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:09.868043 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:09.868010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:11.919638 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:11.919601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:12.001887 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.001857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:12.224106 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.224072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:12.517982 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.517892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:12.613050 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613008    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"]
Jul 24 17:56:12.613441 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613426    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1160d2ea-0c10-4d88-b5c9-295a92e6f71f" containerName="payload-pre-processing"
Jul 24 17:56:12.613490 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613444    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1160d2ea-0c10-4d88-b5c9-295a92e6f71f" containerName="payload-pre-processing"
Jul 24 17:56:12.613490 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613456    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83206685-93b9-446a-ad4c-cf1c4567672a" containerName="revoke-keys"
Jul 24 17:56:12.613490 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613461    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83206685-93b9-446a-ad4c-cf1c4567672a" containerName="revoke-keys"
Jul 24 17:56:12.613490 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613470    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db68d3e-5527-4e19-9599-d47b1eaff4a2" containerName="payload-processing"
Jul 24 17:56:12.613490 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613476    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db68d3e-5527-4e19-9599-d47b1eaff4a2" containerName="payload-processing"
Jul 24 17:56:12.613708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613558    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83206685-93b9-446a-ad4c-cf1c4567672a" containerName="revoke-keys"
Jul 24 17:56:12.613708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613566    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1160d2ea-0c10-4d88-b5c9-295a92e6f71f" containerName="payload-pre-processing"
Jul 24 17:56:12.613708 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.613575    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db68d3e-5527-4e19-9599-d47b1eaff4a2" containerName="payload-processing"
Jul 24 17:56:12.618351 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.618331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"
Jul 24 17:56:12.620682 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.620664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8llqd\""
Jul 24 17:56:12.627306 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.627277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"]
Jul 24 17:56:12.762524 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.762487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ad5008-a911-48cc-8f03-971b81712073-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg\" (UID: \"80ad5008-a911-48cc-8f03-971b81712073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"
Jul 24 17:56:12.863961 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.863862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ad5008-a911-48cc-8f03-971b81712073-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg\" (UID: \"80ad5008-a911-48cc-8f03-971b81712073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"
Jul 24 17:56:12.864623 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.864600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ad5008-a911-48cc-8f03-971b81712073-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg\" (UID: \"80ad5008-a911-48cc-8f03-971b81712073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"
Jul 24 17:56:12.930053 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:12.930004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"
Jul 24 17:56:13.070678 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:13.070621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"]
Jul 24 17:56:13.072537 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:56:13.072508    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ad5008_a911_48cc_8f03_971b81712073.slice/crio-724a8230ab3e27471a69af880be6e14c0d704fa5e67478e2501affd46ef5d270 WatchSource:0}: Error finding container 724a8230ab3e27471a69af880be6e14c0d704fa5e67478e2501affd46ef5d270: Status 404 returned error can't find the container with id 724a8230ab3e27471a69af880be6e14c0d704fa5e67478e2501affd46ef5d270
Jul 24 17:56:14.021330 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:14.021295    2563 generic.go:356] "Generic (PLEG): container finished" podID="80ad5008-a911-48cc-8f03-971b81712073" containerID="c735526a67a89fc54954699bb065cd8f84ba674be94af5e4029d3509579a58d7" exitCode=0
Jul 24 17:56:14.021723 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:14.021339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg" event={"ID":"80ad5008-a911-48cc-8f03-971b81712073","Type":"ContainerDied","Data":"c735526a67a89fc54954699bb065cd8f84ba674be94af5e4029d3509579a58d7"}
Jul 24 17:56:14.021723 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:14.021379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg" event={"ID":"80ad5008-a911-48cc-8f03-971b81712073","Type":"ContainerStarted","Data":"724a8230ab3e27471a69af880be6e14c0d704fa5e67478e2501affd46ef5d270"}
Jul 24 17:56:15.164536 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:15.164507    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"
Jul 24 17:56:15.283783 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:15.283688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ad5008-a911-48cc-8f03-971b81712073-maas-api-service-ca\") pod \"80ad5008-a911-48cc-8f03-971b81712073\" (UID: \"80ad5008-a911-48cc-8f03-971b81712073\") "
Jul 24 17:56:15.284107 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:15.284068    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ad5008-a911-48cc-8f03-971b81712073-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80ad5008-a911-48cc-8f03-971b81712073" (UID: "80ad5008-a911-48cc-8f03-971b81712073"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:56:15.384621 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:15.384586    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ad5008-a911-48cc-8f03-971b81712073-maas-api-service-ca\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:16.032397 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:16.032364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg" event={"ID":"80ad5008-a911-48cc-8f03-971b81712073","Type":"ContainerDied","Data":"724a8230ab3e27471a69af880be6e14c0d704fa5e67478e2501affd46ef5d270"}
Jul 24 17:56:16.032397 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:16.032395    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"
Jul 24 17:56:16.032666 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:16.032412    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="724a8230ab3e27471a69af880be6e14c0d704fa5e67478e2501affd46ef5d270"
Jul 24 17:56:23.283784 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.283747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:23.332299 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.332268    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"]
Jul 24 17:56:23.332546 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.332521    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz" podUID="e41f9704-6dad-4b1a-903a-a4c8f3cfadfe" containerName="payload-processing" containerID="cri-o://d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a" gracePeriod=30
Jul 24 17:56:23.344219 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.344183    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"]
Jul 24 17:56:23.344442 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.344419    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq" podUID="56cf44cd-e826-488b-83ee-11abfdb1a613" containerName="payload-pre-processing" containerID="cri-o://0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2" gracePeriod=30
Jul 24 17:56:23.391425 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.391392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:23.490514 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.490483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:23.606374 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.606345    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:56:23.616796 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.616772    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:56:23.655701 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.655670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cf44cd-e826-488b-83ee-11abfdb1a613-plugins-config-volume\") pod \"56cf44cd-e826-488b-83ee-11abfdb1a613\" (UID: \"56cf44cd-e826-488b-83ee-11abfdb1a613\") "
Jul 24 17:56:23.655896 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.655713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl9lx\" (UniqueName: \"kubernetes.io/projected/56cf44cd-e826-488b-83ee-11abfdb1a613-kube-api-access-jl9lx\") pod \"56cf44cd-e826-488b-83ee-11abfdb1a613\" (UID: \"56cf44cd-e826-488b-83ee-11abfdb1a613\") "
Jul 24 17:56:23.655896 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.655731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-plugins-config-volume\") pod \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\" (UID: \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\") "
Jul 24 17:56:23.655896 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.655748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdm7t\" (UniqueName: \"kubernetes.io/projected/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-kube-api-access-cdm7t\") pod \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\" (UID: \"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe\") "
Jul 24 17:56:23.656133 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.656108    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cf44cd-e826-488b-83ee-11abfdb1a613-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56cf44cd-e826-488b-83ee-11abfdb1a613" (UID: "56cf44cd-e826-488b-83ee-11abfdb1a613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:56:23.656191 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.656113    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e41f9704-6dad-4b1a-903a-a4c8f3cfadfe" (UID: "e41f9704-6dad-4b1a-903a-a4c8f3cfadfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:56:23.658091 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.658065    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cf44cd-e826-488b-83ee-11abfdb1a613-kube-api-access-jl9lx" (OuterVolumeSpecName: "kube-api-access-jl9lx") pod "56cf44cd-e826-488b-83ee-11abfdb1a613" (UID: "56cf44cd-e826-488b-83ee-11abfdb1a613"). InnerVolumeSpecName "kube-api-access-jl9lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:56:23.658177 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.658117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-kube-api-access-cdm7t" (OuterVolumeSpecName: "kube-api-access-cdm7t") pod "e41f9704-6dad-4b1a-903a-a4c8f3cfadfe" (UID: "e41f9704-6dad-4b1a-903a-a4c8f3cfadfe"). InnerVolumeSpecName "kube-api-access-cdm7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:56:23.758106 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.758066    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl9lx\" (UniqueName: \"kubernetes.io/projected/56cf44cd-e826-488b-83ee-11abfdb1a613-kube-api-access-jl9lx\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:23.758106 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.758102    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:23.758372 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.758117    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdm7t\" (UniqueName: \"kubernetes.io/projected/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe-kube-api-access-cdm7t\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:23.758372 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:23.758132    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cf44cd-e826-488b-83ee-11abfdb1a613-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:24.073029 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.072992    2563 generic.go:356] "Generic (PLEG): container finished" podID="e41f9704-6dad-4b1a-903a-a4c8f3cfadfe" containerID="d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a" exitCode=0
Jul 24 17:56:24.073231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.073026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz" event={"ID":"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe","Type":"ContainerDied","Data":"d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a"}
Jul 24 17:56:24.073231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.073067    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"
Jul 24 17:56:24.073231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.073089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz" event={"ID":"e41f9704-6dad-4b1a-903a-a4c8f3cfadfe","Type":"ContainerDied","Data":"3a6e38e3c9131f52799223c656f2eeb13bcbae4766288d87cb056d67412b3340"}
Jul 24 17:56:24.073231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.073117    2563 scope.go:117] "RemoveContainer" containerID="d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a"
Jul 24 17:56:24.074496 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.074475    2563 generic.go:356] "Generic (PLEG): container finished" podID="56cf44cd-e826-488b-83ee-11abfdb1a613" containerID="0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2" exitCode=0
Jul 24 17:56:24.074666 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.074514    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"
Jul 24 17:56:24.074666 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.074539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq" event={"ID":"56cf44cd-e826-488b-83ee-11abfdb1a613","Type":"ContainerDied","Data":"0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2"}
Jul 24 17:56:24.074666 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.074573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq" event={"ID":"56cf44cd-e826-488b-83ee-11abfdb1a613","Type":"ContainerDied","Data":"d7e5c4a780c0b366fbbf1ac860d9e12b675e449f3077b51d6c1ad5e1d3a21907"}
Jul 24 17:56:24.082835 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.082601    2563 scope.go:117] "RemoveContainer" containerID="d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a"
Jul 24 17:56:24.083117 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:56:24.083089    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a\": container with ID starting with d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a not found: ID does not exist" containerID="d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a"
Jul 24 17:56:24.083228 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.083127    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a"} err="failed to get container status \"d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a\": rpc error: code = NotFound desc = could not find container \"d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a\": container with ID starting with d941fb7d7ca831cb6b830064cdbd42672ef51b35a11fc9bc22f2f3e504a0be4a not found: ID does not exist"
Jul 24 17:56:24.083228 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.083145    2563 scope.go:117] "RemoveContainer" containerID="0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2"
Jul 24 17:56:24.091419 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.091398    2563 scope.go:117] "RemoveContainer" containerID="0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2"
Jul 24 17:56:24.091694 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:56:24.091672    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2\": container with ID starting with 0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2 not found: ID does not exist" containerID="0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2"
Jul 24 17:56:24.091791 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.091699    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2"} err="failed to get container status \"0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2\": rpc error: code = NotFound desc = could not find container \"0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2\": container with ID starting with 0d68e4bae0faa40273fae6daa3754b0faa6f6350a87d7a4127212e5f703fd2d2 not found: ID does not exist"
Jul 24 17:56:24.099380 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.099355    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"]
Jul 24 17:56:24.104040 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.104015    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-13c2c7-7fcff7948-j8zrq"]
Jul 24 17:56:24.114657 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.114624    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"]
Jul 24 17:56:24.117050 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.117028    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-13c2c7-b9974c8cd-2lgkz"]
Jul 24 17:56:24.689210 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.689131    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56cf44cd-e826-488b-83ee-11abfdb1a613" path="/var/lib/kubelet/pods/56cf44cd-e826-488b-83ee-11abfdb1a613/volumes"
Jul 24 17:56:24.689559 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:24.689475    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e41f9704-6dad-4b1a-903a-a4c8f3cfadfe" path="/var/lib/kubelet/pods/e41f9704-6dad-4b1a-903a-a4c8f3cfadfe/volumes"
Jul 24 17:56:27.716904 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:27.716865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:27.717278 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:27.716931    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"]
Jul 24 17:56:27.719767 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:27.719747    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13c2c7-ssgpg"]
Jul 24 17:56:28.136798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:28.136707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:28.688092 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:28.688049    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ad5008-a911-48cc-8f03-971b81712073" path="/var/lib/kubelet/pods/80ad5008-a911-48cc-8f03-971b81712073/volumes"
Jul 24 17:56:31.297019 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:31.296984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:34.426107 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:34.426079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:34.528394 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:34.528337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:34.758598 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:34.758540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:35.026545 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.026449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:35.162774 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.162741    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"]
Jul 24 17:56:35.163192 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.163174    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41f9704-6dad-4b1a-903a-a4c8f3cfadfe" containerName="payload-processing"
Jul 24 17:56:35.163192 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.163190    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41f9704-6dad-4b1a-903a-a4c8f3cfadfe" containerName="payload-processing"
Jul 24 17:56:35.163359 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.163197    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ad5008-a911-48cc-8f03-971b81712073" containerName="revoke-keys"
Jul 24 17:56:35.163359 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.163203    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ad5008-a911-48cc-8f03-971b81712073" containerName="revoke-keys"
Jul 24 17:56:35.163359 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.163212    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56cf44cd-e826-488b-83ee-11abfdb1a613" containerName="payload-pre-processing"
Jul 24 17:56:35.163359 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.163218    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56cf44cd-e826-488b-83ee-11abfdb1a613" containerName="payload-pre-processing"
Jul 24 17:56:35.163359 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.163274    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56cf44cd-e826-488b-83ee-11abfdb1a613" containerName="payload-pre-processing"
Jul 24 17:56:35.163359 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.163280    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ad5008-a911-48cc-8f03-971b81712073" containerName="revoke-keys"
Jul 24 17:56:35.163359 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.163288    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e41f9704-6dad-4b1a-903a-a4c8f3cfadfe" containerName="payload-processing"
Jul 24 17:56:35.166382 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.166366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"
Jul 24 17:56:35.172231 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.172212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8llqd\""
Jul 24 17:56:35.180239 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.180213    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"]
Jul 24 17:56:35.254912 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.254873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05117a85-1bca-4918-9e0b-3af30b282909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b9a122-978xl\" (UID: \"05117a85-1bca-4918-9e0b-3af30b282909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"
Jul 24 17:56:35.355534 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.355435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05117a85-1bca-4918-9e0b-3af30b282909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b9a122-978xl\" (UID: \"05117a85-1bca-4918-9e0b-3af30b282909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"
Jul 24 17:56:35.356075 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.356054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05117a85-1bca-4918-9e0b-3af30b282909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b9a122-978xl\" (UID: \"05117a85-1bca-4918-9e0b-3af30b282909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"
Jul 24 17:56:35.476460 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.476421    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"
Jul 24 17:56:35.612714 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:35.612686    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"]
Jul 24 17:56:35.615161 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:56:35.615132    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05117a85_1bca_4918_9e0b_3af30b282909.slice/crio-af53b950d0a7e6ee02bd188c290fdbd668053a8187289041428d9aaf8e6aca14 WatchSource:0}: Error finding container af53b950d0a7e6ee02bd188c290fdbd668053a8187289041428d9aaf8e6aca14: Status 404 returned error can't find the container with id af53b950d0a7e6ee02bd188c290fdbd668053a8187289041428d9aaf8e6aca14
Jul 24 17:56:36.135073 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:36.135039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl" event={"ID":"05117a85-1bca-4918-9e0b-3af30b282909","Type":"ContainerStarted","Data":"d8b6498d94aca0e55a57e0dc32784f5cea27ea645cd7996151c55269133e7515"}
Jul 24 17:56:36.135264 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:36.135106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl" event={"ID":"05117a85-1bca-4918-9e0b-3af30b282909","Type":"ContainerStarted","Data":"af53b950d0a7e6ee02bd188c290fdbd668053a8187289041428d9aaf8e6aca14"}
Jul 24 17:56:36.155984 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:36.155890    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl" podStartSLOduration=1.155874926 podStartE2EDuration="1.155874926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:56:36.154940681 +0000 UTC m=+3199.986193369" watchObservedRunningTime="2026-07-24 17:56:36.155874926 +0000 UTC m=+3199.987127664"
Jul 24 17:56:37.141234 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:37.141161    2563 generic.go:356] "Generic (PLEG): container finished" podID="05117a85-1bca-4918-9e0b-3af30b282909" containerID="d8b6498d94aca0e55a57e0dc32784f5cea27ea645cd7996151c55269133e7515" exitCode=0
Jul 24 17:56:37.141868 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:37.141223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl" event={"ID":"05117a85-1bca-4918-9e0b-3af30b282909","Type":"ContainerDied","Data":"d8b6498d94aca0e55a57e0dc32784f5cea27ea645cd7996151c55269133e7515"}
Jul 24 17:56:38.286150 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:38.286125    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"
Jul 24 17:56:38.383837 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:38.383804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05117a85-1bca-4918-9e0b-3af30b282909-maas-api-service-ca\") pod \"05117a85-1bca-4918-9e0b-3af30b282909\" (UID: \"05117a85-1bca-4918-9e0b-3af30b282909\") "
Jul 24 17:56:38.384148 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:38.384128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05117a85-1bca-4918-9e0b-3af30b282909-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05117a85-1bca-4918-9e0b-3af30b282909" (UID: "05117a85-1bca-4918-9e0b-3af30b282909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:56:38.484972 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:38.484940    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05117a85-1bca-4918-9e0b-3af30b282909-maas-api-service-ca\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:39.152025 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:39.151994    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"
Jul 24 17:56:39.152025 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:39.152016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl" event={"ID":"05117a85-1bca-4918-9e0b-3af30b282909","Type":"ContainerDied","Data":"af53b950d0a7e6ee02bd188c290fdbd668053a8187289041428d9aaf8e6aca14"}
Jul 24 17:56:39.152263 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:39.152060    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af53b950d0a7e6ee02bd188c290fdbd668053a8187289041428d9aaf8e6aca14"
Jul 24 17:56:45.703526 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:45.703495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:45.719566 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:45.719533    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"]
Jul 24 17:56:45.719854 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:45.719828    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm" podUID="29a79bbb-6304-4243-a20a-b862a7f2bb26" containerName="payload-processing" containerID="cri-o://818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478" gracePeriod=30
Jul 24 17:56:45.728340 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:45.728293    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"]
Jul 24 17:56:45.729633 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:45.728604    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg" podUID="5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00" containerName="payload-pre-processing" containerID="cri-o://31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56" gracePeriod=30
Jul 24 17:56:45.803017 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:45.802974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:46.004640 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.004615    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:56:46.007198 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.007180    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:56:46.051272 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.051240    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a79bbb-6304-4243-a20a-b862a7f2bb26-plugins-config-volume\") pod \"29a79bbb-6304-4243-a20a-b862a7f2bb26\" (UID: \"29a79bbb-6304-4243-a20a-b862a7f2bb26\") "
Jul 24 17:56:46.051467 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.051346    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-plugins-config-volume\") pod \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\" (UID: \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\") "
Jul 24 17:56:46.051467 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.051379    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvknq\" (UniqueName: \"kubernetes.io/projected/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-kube-api-access-gvknq\") pod \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\" (UID: \"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00\") "
Jul 24 17:56:46.051467 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.051409    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpr9c\" (UniqueName: \"kubernetes.io/projected/29a79bbb-6304-4243-a20a-b862a7f2bb26-kube-api-access-vpr9c\") pod \"29a79bbb-6304-4243-a20a-b862a7f2bb26\" (UID: \"29a79bbb-6304-4243-a20a-b862a7f2bb26\") "
Jul 24 17:56:46.051787 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.051757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a79bbb-6304-4243-a20a-b862a7f2bb26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29a79bbb-6304-4243-a20a-b862a7f2bb26" (UID: "29a79bbb-6304-4243-a20a-b862a7f2bb26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:56:46.051867 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.051794    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00" (UID: "5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:56:46.053826 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.053802    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-kube-api-access-gvknq" (OuterVolumeSpecName: "kube-api-access-gvknq") pod "5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00" (UID: "5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00"). InnerVolumeSpecName "kube-api-access-gvknq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:56:46.053826 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.053828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a79bbb-6304-4243-a20a-b862a7f2bb26-kube-api-access-vpr9c" (OuterVolumeSpecName: "kube-api-access-vpr9c") pod "29a79bbb-6304-4243-a20a-b862a7f2bb26" (UID: "29a79bbb-6304-4243-a20a-b862a7f2bb26"). InnerVolumeSpecName "kube-api-access-vpr9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:56:46.152860 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.152823    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a79bbb-6304-4243-a20a-b862a7f2bb26-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:46.152860 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.152853    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-plugins-config-volume\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:46.152860 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.152865    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvknq\" (UniqueName: \"kubernetes.io/projected/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00-kube-api-access-gvknq\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:46.153100 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.152873    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpr9c\" (UniqueName: \"kubernetes.io/projected/29a79bbb-6304-4243-a20a-b862a7f2bb26-kube-api-access-vpr9c\") on node \"ip-10-0-140-80.ec2.internal\" DevicePath \"\""
Jul 24 17:56:46.191798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.191762    2563 generic.go:356] "Generic (PLEG): container finished" podID="29a79bbb-6304-4243-a20a-b862a7f2bb26" containerID="818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478" exitCode=0
Jul 24 17:56:46.191994 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.191820    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"
Jul 24 17:56:46.191994 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.191846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm" event={"ID":"29a79bbb-6304-4243-a20a-b862a7f2bb26","Type":"ContainerDied","Data":"818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478"}
Jul 24 17:56:46.191994 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.191936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm" event={"ID":"29a79bbb-6304-4243-a20a-b862a7f2bb26","Type":"ContainerDied","Data":"b8f79de4466649881b47d786f18675d2591acd58b3d5cf8df89d07a236dd6a8a"}
Jul 24 17:56:46.191994 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.191966    2563 scope.go:117] "RemoveContainer" containerID="818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478"
Jul 24 17:56:46.193348 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.193321    2563 generic.go:356] "Generic (PLEG): container finished" podID="5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00" containerID="31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56" exitCode=0
Jul 24 17:56:46.193449 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.193368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg" event={"ID":"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00","Type":"ContainerDied","Data":"31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56"}
Jul 24 17:56:46.193449 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.193391    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"
Jul 24 17:56:46.193449 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.193405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg" event={"ID":"5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00","Type":"ContainerDied","Data":"d6469060c4da534e4fa5dc323d1fbbbe31cc0be6df524ee373f66545da22f9d0"}
Jul 24 17:56:46.201167 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.201148    2563 scope.go:117] "RemoveContainer" containerID="818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478"
Jul 24 17:56:46.201443 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:56:46.201423    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478\": container with ID starting with 818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478 not found: ID does not exist" containerID="818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478"
Jul 24 17:56:46.201520 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.201454    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478"} err="failed to get container status \"818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478\": rpc error: code = NotFound desc = could not find container \"818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478\": container with ID starting with 818bc3d6e0e932679af3a9b3d57d30fdfd59a9122b8c7f9337297b81d2623478 not found: ID does not exist"
Jul 24 17:56:46.201520 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.201477    2563 scope.go:117] "RemoveContainer" containerID="31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56"
Jul 24 17:56:46.209873 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.209855    2563 scope.go:117] "RemoveContainer" containerID="31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56"
Jul 24 17:56:46.210190 ip-10-0-140-80 kubenswrapper[2563]: E0724 17:56:46.210167    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56\": container with ID starting with 31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56 not found: ID does not exist" containerID="31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56"
Jul 24 17:56:46.210273 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.210200    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56"} err="failed to get container status \"31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56\": rpc error: code = NotFound desc = could not find container \"31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56\": container with ID starting with 31b94256603e012b7a4093c92e61019bd265a897c24105a3d42abcf4e92dcb56 not found: ID does not exist"
Jul 24 17:56:46.223609 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.223581    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"]
Jul 24 17:56:46.226798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.226737    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-b9a122-79b6bd7478-h9npm"]
Jul 24 17:56:46.236808 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.236783    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"]
Jul 24 17:56:46.239419 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.239396    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-b9a122-68965657bd-bjsqg"]
Jul 24 17:56:46.689967 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.689891    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a79bbb-6304-4243-a20a-b862a7f2bb26" path="/var/lib/kubelet/pods/29a79bbb-6304-4243-a20a-b862a7f2bb26/volumes"
Jul 24 17:56:46.690254 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:46.690242    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00" path="/var/lib/kubelet/pods/5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00/volumes"
Jul 24 17:56:50.243861 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:50.243826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:50.250793 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:50.250769    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"]
Jul 24 17:56:50.256729 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:50.256706    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b9a122-978xl"]
Jul 24 17:56:50.688487 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:50.688392    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05117a85-1bca-4918-9e0b-3af30b282909" path="/var/lib/kubelet/pods/05117a85-1bca-4918-9e0b-3af30b282909/volumes"
Jul 24 17:56:50.764944 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:50.764915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:56.955726 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:56.955690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:57.091020 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:57.090982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:57.263230 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:57.263200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:56:57.525851 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:56:57.525763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:57:04.198393 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:04.198361    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-87594f4b5-wfjwj_b7530b11-63ea-47af-a4f9-abcb2150c240/manager/0.log"
Jul 24 17:57:04.904308 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:04.904276    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-645867896-kxdff_4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0/manager/0.log"
Jul 24 17:57:04.914500 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:04.914475    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-645867896-kxdff_4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0/copy-manifests/0.log"
Jul 24 17:57:05.547474 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:05.547449    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748585-24kqh_536d23dc-8247-49e3-9cf0-0a0e2a35e033/cleanup/0.log"
Jul 24 17:57:05.668505 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:05.668473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-xmjxf_2d74ad54-5972-4531-860a-857d62cff6b2/postgres/0.log"
Jul 24 17:57:07.513995 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:07.513956    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kszgh_21904cd6-c74f-44e2-9ff5-c0f67a7e61cc/kuadrant-console-plugin/0.log"
Jul 24 17:57:08.800993 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:08.800953    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8cdf7544b-lwnng_50f9199d-0dab-4be8-9021-3089cdd759da/kube-auth-proxy/0.log"
Jul 24 17:57:09.069967 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:09.069874    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zbfnf_2ba6c789-6276-471c-8198-999aa9dfab2a/payload-pre-processing/0.log"
Jul 24 17:57:09.201204 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:09.201174    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8n49r_9af7f2dd-29f5-43b5-964f-95142729e455/payload-processing/0.log"
Jul 24 17:57:09.830630 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:09.830586    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-qtlkw_b490414c-4d11-4035-a522-7c5c869098b0/main/0.log"
Jul 24 17:57:09.972262 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:09.972238    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-m495g_3a11148e-837d-4b22-9b37-9b55b52b4714/main/0.log"
Jul 24 17:57:16.205484 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.205447    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sr8cd/must-gather-scvjs"]
Jul 24 17:57:16.205905 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.205860    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a79bbb-6304-4243-a20a-b862a7f2bb26" containerName="payload-processing"
Jul 24 17:57:16.205905 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.205876    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a79bbb-6304-4243-a20a-b862a7f2bb26" containerName="payload-processing"
Jul 24 17:57:16.205905 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.205894    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00" containerName="payload-pre-processing"
Jul 24 17:57:16.205905 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.205902    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00" containerName="payload-pre-processing"
Jul 24 17:57:16.206050 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.205914    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05117a85-1bca-4918-9e0b-3af30b282909" containerName="revoke-keys"
Jul 24 17:57:16.206050 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.205919    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05117a85-1bca-4918-9e0b-3af30b282909" containerName="revoke-keys"
Jul 24 17:57:16.206050 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.206027    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05117a85-1bca-4918-9e0b-3af30b282909" containerName="revoke-keys"
Jul 24 17:57:16.206050 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.206041    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a95ef01-847e-4ad4-ae3e-89a7c9a1cf00" containerName="payload-pre-processing"
Jul 24 17:57:16.206198 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.206053    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a79bbb-6304-4243-a20a-b862a7f2bb26" containerName="payload-processing"
Jul 24 17:57:16.209377 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.209355    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sr8cd/must-gather-scvjs"
Jul 24 17:57:16.212003 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.211975    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sr8cd\"/\"kube-root-ca.crt\""
Jul 24 17:57:16.212773 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.212751    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sr8cd\"/\"openshift-service-ca.crt\""
Jul 24 17:57:16.212773 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.212766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sr8cd\"/\"default-dockercfg-6zpqp\""
Jul 24 17:57:16.221692 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.221635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sr8cd/must-gather-scvjs"]
Jul 24 17:57:16.228369 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.228343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/851b04b7-a97c-47b3-9fcf-d30fbb17d587-must-gather-output\") pod \"must-gather-scvjs\" (UID: \"851b04b7-a97c-47b3-9fcf-d30fbb17d587\") " pod="openshift-must-gather-sr8cd/must-gather-scvjs"
Jul 24 17:57:16.228466 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.228408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvc7n\" (UniqueName: \"kubernetes.io/projected/851b04b7-a97c-47b3-9fcf-d30fbb17d587-kube-api-access-wvc7n\") pod \"must-gather-scvjs\" (UID: \"851b04b7-a97c-47b3-9fcf-d30fbb17d587\") " pod="openshift-must-gather-sr8cd/must-gather-scvjs"
Jul 24 17:57:16.329001 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.328967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/851b04b7-a97c-47b3-9fcf-d30fbb17d587-must-gather-output\") pod \"must-gather-scvjs\" (UID: \"851b04b7-a97c-47b3-9fcf-d30fbb17d587\") " pod="openshift-must-gather-sr8cd/must-gather-scvjs"
Jul 24 17:57:16.329186 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.329030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvc7n\" (UniqueName: \"kubernetes.io/projected/851b04b7-a97c-47b3-9fcf-d30fbb17d587-kube-api-access-wvc7n\") pod \"must-gather-scvjs\" (UID: \"851b04b7-a97c-47b3-9fcf-d30fbb17d587\") " pod="openshift-must-gather-sr8cd/must-gather-scvjs"
Jul 24 17:57:16.329308 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.329290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/851b04b7-a97c-47b3-9fcf-d30fbb17d587-must-gather-output\") pod \"must-gather-scvjs\" (UID: \"851b04b7-a97c-47b3-9fcf-d30fbb17d587\") " pod="openshift-must-gather-sr8cd/must-gather-scvjs"
Jul 24 17:57:16.337224 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.337196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvc7n\" (UniqueName: \"kubernetes.io/projected/851b04b7-a97c-47b3-9fcf-d30fbb17d587-kube-api-access-wvc7n\") pod \"must-gather-scvjs\" (UID: \"851b04b7-a97c-47b3-9fcf-d30fbb17d587\") " pod="openshift-must-gather-sr8cd/must-gather-scvjs"
Jul 24 17:57:16.523968 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.523941    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sr8cd\"/\"default-dockercfg-6zpqp\""
Jul 24 17:57:16.530101 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.530080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sr8cd/must-gather-scvjs"
Jul 24 17:57:16.670307 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.670270    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sr8cd/must-gather-scvjs"]
Jul 24 17:57:16.672547 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:57:16.672521    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod851b04b7_a97c_47b3_9fcf_d30fbb17d587.slice/crio-b988aeaa992a964a5f9fa0b94e2eb75270b2ebb565ce6d4d76466842273adccf WatchSource:0}: Error finding container b988aeaa992a964a5f9fa0b94e2eb75270b2ebb565ce6d4d76466842273adccf: Status 404 returned error can't find the container with id b988aeaa992a964a5f9fa0b94e2eb75270b2ebb565ce6d4d76466842273adccf
Jul 24 17:57:16.674398 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:16.674384    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:57:17.087000 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:17.086966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:57:17.187194 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:17.187161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log" limit=1024
Jul 24 17:57:17.352172 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:17.352082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sr8cd/must-gather-scvjs" event={"ID":"851b04b7-a97c-47b3-9fcf-d30fbb17d587","Type":"ContainerStarted","Data":"b988aeaa992a964a5f9fa0b94e2eb75270b2ebb565ce6d4d76466842273adccf"}
Jul 24 17:57:18.360146 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:18.360092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sr8cd/must-gather-scvjs" event={"ID":"851b04b7-a97c-47b3-9fcf-d30fbb17d587","Type":"ContainerStarted","Data":"af2dd8bdb820cf744135e403876e642991b8ab81ff218bff6c713fa8bab71f92"}
Jul 24 17:57:18.360576 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:18.360166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sr8cd/must-gather-scvjs" event={"ID":"851b04b7-a97c-47b3-9fcf-d30fbb17d587","Type":"ContainerStarted","Data":"215632696ef6553f1c8fc80a6574de6dfcbb8f3d3a3bda2e1030b0b066d98ec0"}
Jul 24 17:57:18.378764 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:18.378714    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sr8cd/must-gather-scvjs" podStartSLOduration=1.546347943 podStartE2EDuration="2.378697238s" totalImagesPullingTime="832.349295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:57:18.377727485 +0000 UTC m=+3242.208980172" watchObservedRunningTime="2026-07-24 17:57:18.378697238 +0000 UTC m=+3242.209949925"
Jul 24 17:57:19.182183 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:19.182154    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-58fw9_da9a090f-6bf3-4c56-87c9-c48b9fa615d4/global-pull-secret-syncer/0.log"
Jul 24 17:57:19.282171 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:19.282134    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dpxpq_02ca41c2-e80b-4342-bc4b-93b2ccf4ed0c/konnectivity-agent/0.log"
Jul 24 17:57:19.436051 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:19.435968    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-80.ec2.internal_3526c73b5b056c8ae8b4457c3416ed52/haproxy/0.log"
Jul 24 17:57:25.380531 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:25.380495    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kszgh_21904cd6-c74f-44e2-9ff5-c0f67a7e61cc/kuadrant-console-plugin/0.log"
Jul 24 17:57:27.330751 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.330631    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5e40b7a7-a81d-4283-857b-253a273d952a/alertmanager/0.log"
Jul 24 17:57:27.364082 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.364051    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5e40b7a7-a81d-4283-857b-253a273d952a/config-reloader/0.log"
Jul 24 17:57:27.401065 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.401024    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5e40b7a7-a81d-4283-857b-253a273d952a/kube-rbac-proxy-web/0.log"
Jul 24 17:57:27.444878 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.444840    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5e40b7a7-a81d-4283-857b-253a273d952a/kube-rbac-proxy/0.log"
Jul 24 17:57:27.485103 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.485075    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5e40b7a7-a81d-4283-857b-253a273d952a/kube-rbac-proxy-metric/0.log"
Jul 24 17:57:27.526252 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.526226    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5e40b7a7-a81d-4283-857b-253a273d952a/prom-label-proxy/0.log"
Jul 24 17:57:27.562002 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.561966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5e40b7a7-a81d-4283-857b-253a273d952a/init-config-reloader/0.log"
Jul 24 17:57:27.926616 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.926587    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ns2w_8b62d8c1-cb04-4645-9684-793f2a261d22/node-exporter/0.log"
Jul 24 17:57:27.957463 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.957434    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ns2w_8b62d8c1-cb04-4645-9684-793f2a261d22/kube-rbac-proxy/0.log"
Jul 24 17:57:27.993823 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:27.993788    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ns2w_8b62d8c1-cb04-4645-9684-793f2a261d22/init-textfile/0.log"
Jul 24 17:57:28.138753 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.138723    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-7lz8q_ff207798-7666-45d2-84fa-3737f4e6ad65/kube-rbac-proxy-main/0.log"
Jul 24 17:57:28.163979 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.163951    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-7lz8q_ff207798-7666-45d2-84fa-3737f4e6ad65/kube-rbac-proxy-self/0.log"
Jul 24 17:57:28.188267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.188194    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-7lz8q_ff207798-7666-45d2-84fa-3737f4e6ad65/openshift-state-metrics/0.log"
Jul 24 17:57:28.230580 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.230551    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8889927-f4b8-4b6b-93f6-50fe15fef290/prometheus/0.log"
Jul 24 17:57:28.260304 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.260276    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8889927-f4b8-4b6b-93f6-50fe15fef290/config-reloader/0.log"
Jul 24 17:57:28.288702 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.288671    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8889927-f4b8-4b6b-93f6-50fe15fef290/thanos-sidecar/0.log"
Jul 24 17:57:28.313628 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.313601    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8889927-f4b8-4b6b-93f6-50fe15fef290/kube-rbac-proxy-web/0.log"
Jul 24 17:57:28.346280 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.346255    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8889927-f4b8-4b6b-93f6-50fe15fef290/kube-rbac-proxy/0.log"
Jul 24 17:57:28.372785 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.372759    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8889927-f4b8-4b6b-93f6-50fe15fef290/kube-rbac-proxy-thanos/0.log"
Jul 24 17:57:28.399087 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.399047    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8889927-f4b8-4b6b-93f6-50fe15fef290/init-config-reloader/0.log"
Jul 24 17:57:28.430610 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.430580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-d9cjn_bf21d174-fa96-4af3-b381-317d9415785d/prometheus-operator/0.log"
Jul 24 17:57:28.450673 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.450554    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-d9cjn_bf21d174-fa96-4af3-b381-317d9415785d/kube-rbac-proxy/0.log"
Jul 24 17:57:28.478434 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:28.478405    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-2gfmq_42611f4e-4bca-4b5e-b6f0-e68314d12922/prometheus-operator-admission-webhook/0.log"
Jul 24 17:57:29.202129 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.202079    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"]
Jul 24 17:57:29.210054 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.210019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.222218 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.221416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"]
Jul 24 17:57:29.270238 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.270201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-proc\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.270511 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.270490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd52r\" (UniqueName: \"kubernetes.io/projected/b75846ec-d0b2-489e-a9ec-1202200b8e7b-kube-api-access-pd52r\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.270719 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.270701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-sys\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.270883 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.270869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-lib-modules\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.271035 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.271015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-podres\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.372332 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.372289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-proc\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.372798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.372392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-proc\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.372798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.372411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd52r\" (UniqueName: \"kubernetes.io/projected/b75846ec-d0b2-489e-a9ec-1202200b8e7b-kube-api-access-pd52r\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.372798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.372487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-sys\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.372798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.372515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-lib-modules\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.372798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.372540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-podres\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.372798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.372629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-sys\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.372798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.372704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-podres\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.372798 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.372707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b75846ec-d0b2-489e-a9ec-1202200b8e7b-lib-modules\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.382369 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.382330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd52r\" (UniqueName: \"kubernetes.io/projected/b75846ec-d0b2-489e-a9ec-1202200b8e7b-kube-api-access-pd52r\") pod \"perf-node-gather-daemonset-9bb4d\" (UID: \"b75846ec-d0b2-489e-a9ec-1202200b8e7b\") " pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.526567 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.526526    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:29.719780 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:29.719749    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"]
Jul 24 17:57:29.721977 ip-10-0-140-80 kubenswrapper[2563]: W0724 17:57:29.721934    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb75846ec_d0b2_489e_a9ec_1202200b8e7b.slice/crio-51a117df2e298fb097cfb0e49d35d3f6f301b0497efc93411d79aba96d26b4a0 WatchSource:0}: Error finding container 51a117df2e298fb097cfb0e49d35d3f6f301b0497efc93411d79aba96d26b4a0: Status 404 returned error can't find the container with id 51a117df2e298fb097cfb0e49d35d3f6f301b0497efc93411d79aba96d26b4a0
Jul 24 17:57:30.453753 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:30.453715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d" event={"ID":"b75846ec-d0b2-489e-a9ec-1202200b8e7b","Type":"ContainerStarted","Data":"4dac58a7da1452952ec5c61a713b8a4278d3647b698c128039d903b3b90843ca"}
Jul 24 17:57:30.454267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:30.453800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d" event={"ID":"b75846ec-d0b2-489e-a9ec-1202200b8e7b","Type":"ContainerStarted","Data":"51a117df2e298fb097cfb0e49d35d3f6f301b0497efc93411d79aba96d26b4a0"}
Jul 24 17:57:30.454267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:30.453859    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:30.470946 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:30.470886    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d" podStartSLOduration=1.470868495 podStartE2EDuration="1.470868495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:57:30.469065183 +0000 UTC m=+3254.300317893" watchObservedRunningTime="2026-07-24 17:57:30.470868495 +0000 UTC m=+3254.302121181"
Jul 24 17:57:30.974826 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:30.974799    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-r2v9x_99ba6f44-50af-47c9-9fec-370390ebfd14/download-server/0.log"
Jul 24 17:57:32.723475 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:32.723451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zk69n_d8201c68-e268-411e-886e-87a4c1429052/dns/0.log"
Jul 24 17:57:32.756040 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:32.756013    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zk69n_d8201c68-e268-411e-886e-87a4c1429052/kube-rbac-proxy/0.log"
Jul 24 17:57:32.837262 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:32.837226    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mplnn_852e6ad4-88e9-486e-962f-cfc44efa06ca/dns-node-resolver/0.log"
Jul 24 17:57:33.589041 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:33.589016    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztz7f_88c4f897-c3ac-4191-aee8-976fe6a42699/node-ca/0.log"
Jul 24 17:57:34.786625 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:34.786588    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8cdf7544b-lwnng_50f9199d-0dab-4be8-9021-3089cdd759da/kube-auth-proxy/0.log"
Jul 24 17:57:34.912866 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:34.912833    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zbfnf_2ba6c789-6276-471c-8198-999aa9dfab2a/payload-pre-processing/0.log"
Jul 24 17:57:34.954112 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:34.954052    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8n49r_9af7f2dd-29f5-43b5-964f-95142729e455/payload-processing/0.log"
Jul 24 17:57:35.835607 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:35.835572    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w25vs_cec7ee9b-390e-407d-a9a6-7ff6d615867a/serve-healthcheck-canary/0.log"
Jul 24 17:57:36.381458 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:36.381427    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pb8t2_726e53f0-e3d8-4414-929b-683f5f4eb5c2/kube-rbac-proxy/0.log"
Jul 24 17:57:36.420267 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:36.420241    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pb8t2_726e53f0-e3d8-4414-929b-683f5f4eb5c2/exporter/0.log"
Jul 24 17:57:36.447430 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:36.447403    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pb8t2_726e53f0-e3d8-4414-929b-683f5f4eb5c2/extractor/0.log"
Jul 24 17:57:36.473706 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:36.473678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sr8cd/perf-node-gather-daemonset-9bb4d"
Jul 24 17:57:39.625986 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:39.625874    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-87594f4b5-wfjwj_b7530b11-63ea-47af-a4f9-abcb2150c240/manager/0.log"
Jul 24 17:57:40.019606 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:40.019573    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-645867896-kxdff_4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0/manager/0.log"
Jul 24 17:57:40.059859 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:40.059828    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8cdf7544b-lwnng_50f9199d-0dab-4be8-9021-3089cdd759da/kube-auth-proxy/0.log"
Jul 24 17:57:40.072849 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:40.072821    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-645867896-kxdff_4e569e0f-fc4f-466f-81c9-fd9d4ff1d0f0/copy-manifests/0.log"
Jul 24 17:57:40.180042 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:40.180008    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zbfnf_2ba6c789-6276-471c-8198-999aa9dfab2a/payload-pre-processing/0.log"
Jul 24 17:57:40.207335 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:40.207293    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8n49r_9af7f2dd-29f5-43b5-964f-95142729e455/payload-processing/0.log"
Jul 24 17:57:41.367693 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:41.367642    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f7bfdc884-bxnvh_7eb9267f-5cf0-4536-bc3b-a40a775afdb5/manager/0.log"
Jul 24 17:57:41.824804 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:41.824779    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8cdf7544b-lwnng_50f9199d-0dab-4be8-9021-3089cdd759da/kube-auth-proxy/0.log"
Jul 24 17:57:41.932843 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:41.932814    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zbfnf_2ba6c789-6276-471c-8198-999aa9dfab2a/payload-pre-processing/0.log"
Jul 24 17:57:41.959736 ip-10-0-140-80 kubenswrapper[2563]: I0724 17:57:41.959704    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8n49r_9af7f2dd-29f5-43b5-964f-95142729e455/payload-processing/0.log"